2024-06-15T07:24:06.634 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-15T07:24:06.634 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785 --name yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/orig.config.yaml 2024-06-15T07:24:06.669 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T07:24:06.761 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785 branch: wip-yuri5-testing-2024-06-08-0902-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7756785' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi no_nested_subset: false os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri5-testing-2024-06-08-0902-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 rgw-multisite-tests: args: - tests.py workunit: branch: wip-reef-drop-centos8 sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 8659 sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-reef-drop-centos8 suite_path: /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBY/mxvVX7P4IG2Dm/bOwz9MRbKtI9BfQX0UPeVZGLZFeDIHNpYIKG/jJFJzA+qwA+Zwf8C+QeH61+6wTjTpg8= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9YO8YSxAcQpNBx9QAf6bw6L9BwqGOqgW0B3R9qnUbclUzl56KIW9mLr5HJMLZcyc2Xbm5Uw2X2CHr7R2sCq0Y= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-14_18:51:11 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-15T07:24:06.761 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa; will attempt to use it 2024-06-15T07:24:06.762 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks 2024-06-15T07:24:06.762 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-15T07:24:06.763 INFO:teuthology.task.internal:Checking packages... 2024-06-15T07:24:06.786 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '6f4b09560ed399c3f139402a02f389c2fc8bb2cc' 2024-06-15T07:24:06.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T07:24:06.787 INFO:teuthology.packaging:ref: None 2024-06-15T07:24:06.787 INFO:teuthology.packaging:tag: None 2024-06-15T07:24:06.787 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-06-08-0902-reef 2024-06-15T07:24:06.787 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T07:24:06.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri5-testing-2024-06-08-0902-reef 2024-06-15T07:24:07.007 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:24:07.008 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-15T07:24:07.014 INFO:teuthology.task.internal:no buildpackages task found 2024-06-15T07:24:07.014 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-15T07:24:07.045 INFO:teuthology.task.internal:Saving configuration 2024-06-15T07:24:07.058 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-15T07:24:07.104 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-15T07:24:07.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-15 07:18:49.102291', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+HA0+eft3+x9uyWViZwbTFSnBaftUQzgqEi35r1DNK'} 2024-06-15T07:24:07.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-15 07:18:49.103834', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDmCQYgheT/dXTKJz4JVLqpTEXHgdYC/RlHakw2UVQZZ'} 2024-06-15T07:24:07.155 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-15T07:24:07.161 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-06-15T07:24:07.161 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-06-15T07:24:07.161 INFO:teuthology.run_tasks:Running task console_log... 2024-06-15T07:24:07.243 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f36c69b3be0>, signals=[15]) 2024-06-15T07:24:07.243 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-15T07:24:07.249 INFO:teuthology.task.internal:Opening connections... 2024-06-15T07:24:07.249 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:24:07.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:24:07.339 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:24:07.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:24:07.425 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-15T07:24:07.431 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-06-15T07:24:07.436 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-06-15T07:24:07.436 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-06-15T07:24:07.483 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Ubuntu" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="22.04" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_CODENAME=jammy 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:ID=ubuntu 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE=debian 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-15T07:24:07.484 INFO:teuthology.orchestra.run.smithi012.stdout:UBUNTU_CODENAME=jammy 2024-06-15T07:24:07.485 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-06-15T07:24:07.502 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-06-15T07:24:07.507 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-06-15T07:24:07.507 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="Ubuntu" 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="22.04" 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_CODENAME=jammy 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:ID=ubuntu 2024-06-15T07:24:07.552 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE=debian 2024-06-15T07:24:07.553 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-15T07:24:07.553 INFO:teuthology.orchestra.run.smithi131.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-15T07:24:07.553 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-15T07:24:07.553 INFO:teuthology.orchestra.run.smithi131.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-15T07:24:07.553 INFO:teuthology.orchestra.run.smithi131.stdout:UBUNTU_CODENAME=jammy 2024-06-15T07:24:07.553 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-06-15T07:24:07.577 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-15T07:24:07.586 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-15T07:24:07.603 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-15T07:24:07.604 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-06-15T07:24:07.606 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-06-15T07:24:07.610 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-15T07:24:07.623 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-15T07:24:07.623 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-06-15T07:24:07.653 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-06-15T07:24:07.749 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-15T07:24:07.806 INFO:teuthology.run_tasks:Running task kernel... 2024-06-15T07:24:07.820 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-15T07:24:07.821 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-15T07:24:07.821 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-15T07:24:07.821 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-15T07:24:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:24:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:24:07.827 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-06-15T07:24:07.827 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-06-15T07:24:07.873 INFO:teuthology.orchestra.run.smithi131.stdout:5.15.0-56-generic 2024-06-15T07:24:07.873 INFO:teuthology.task.kernel:Running kernel on smithi131: 5.15.0-56-generic 2024-06-15T07:24:07.873 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get clean 2024-06-15T07:24:07.876 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-56-generic 2024-06-15T07:24:07.876 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.15.0-56-generic 2024-06-15T07:24:07.876 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-06-15T07:24:08.001 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2024-06-15T07:24:08.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-15T07:24:08.136 INFO:teuthology.orchestra.run.smithi131.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-15T07:24:08.248 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:24:08.286 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:24:08.295 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-15T07:24:08.332 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-15T07:24:08.350 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-15T07:24:08.369 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,517 kB] 2024-06-15T07:24:08.482 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [480 kB] 2024-06-15T07:24:08.488 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [259 kB] 2024-06-15T07:24:08.490 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-15T07:24:08.491 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-06-15T07:24:08.491 INFO:teuthology.orchestra.run.smithi131.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,933 kB] 2024-06-15T07:24:08.522 INFO:teuthology.orchestra.run.smithi131.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [329 kB] 2024-06-15T07:24:08.526 INFO:teuthology.orchestra.run.smithi131.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T07:24:08.527 INFO:teuthology.orchestra.run.smithi131.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [858 kB] 2024-06-15T07:24:08.536 INFO:teuthology.orchestra.run.smithi131.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [607 kB] 2024-06-15T07:24:08.542 INFO:teuthology.orchestra.run.smithi131.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [166 kB] 2024-06-15T07:24:08.543 INFO:teuthology.orchestra.run.smithi131.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-15T07:24:08.544 INFO:teuthology.orchestra.run.smithi131.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-15T07:24:08.544 INFO:teuthology.orchestra.run.smithi131.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-15T07:24:08.578 INFO:teuthology.orchestra.run.smithi131.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-15T07:24:08.578 INFO:teuthology.orchestra.run.smithi131.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-15T07:24:08.688 INFO:teuthology.orchestra.run.smithi131.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-15T07:24:08.706 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-15T07:24:08.731 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [480 kB] 2024-06-15T07:24:08.826 INFO:teuthology.orchestra.run.smithi131.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [642 kB] 2024-06-15T07:24:08.845 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,730 kB] 2024-06-15T07:24:09.013 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,517 kB] 2024-06-15T07:24:09.015 INFO:teuthology.orchestra.run.smithi131.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,730 kB] 2024-06-15T07:24:09.145 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [642 kB] 2024-06-15T07:24:09.165 INFO:teuthology.orchestra.run.smithi131.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [318 kB] 2024-06-15T07:24:09.179 INFO:teuthology.orchestra.run.smithi131.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-15T07:24:09.180 INFO:teuthology.orchestra.run.smithi131.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,990 kB] 2024-06-15T07:24:09.181 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [259 kB] 2024-06-15T07:24:09.183 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [318 kB] 2024-06-15T07:24:09.193 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-15T07:24:09.194 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-06-15T07:24:09.196 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,933 kB] 2024-06-15T07:24:09.197 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-15T07:24:09.198 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,990 kB] 2024-06-15T07:24:09.244 INFO:teuthology.orchestra.run.smithi131.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-06-15T07:24:09.251 INFO:teuthology.orchestra.run.smithi131.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [338 kB] 2024-06-15T07:24:09.258 INFO:teuthology.orchestra.run.smithi131.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T07:24:09.258 INFO:teuthology.orchestra.run.smithi131.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [707 kB] 2024-06-15T07:24:09.261 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-06-15T07:24:09.262 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [338 kB] 2024-06-15T07:24:09.270 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T07:24:09.270 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,086 kB] 2024-06-15T07:24:09.271 INFO:teuthology.orchestra.run.smithi131.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,086 kB] 2024-06-15T07:24:09.285 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [329 kB] 2024-06-15T07:24:09.292 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T07:24:09.292 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [858 kB] 2024-06-15T07:24:09.293 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [707 kB] 2024-06-15T07:24:09.304 INFO:teuthology.orchestra.run.smithi131.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [251 kB] 2024-06-15T07:24:09.308 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [607 kB] 2024-06-15T07:24:09.308 INFO:teuthology.orchestra.run.smithi131.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-15T07:24:09.318 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [166 kB] 2024-06-15T07:24:09.321 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [251 kB] 2024-06-15T07:24:09.326 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-15T07:24:09.326 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-06-15T07:24:09.327 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.0 kB] 2024-06-15T07:24:09.330 INFO:teuthology.orchestra.run.smithi131.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.0 kB] 2024-06-15T07:24:09.330 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-15T07:24:09.331 INFO:teuthology.orchestra.run.smithi131.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-06-15T07:24:09.347 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-15T07:24:09.348 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-15T07:24:09.348 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-15T07:24:09.389 INFO:teuthology.orchestra.run.smithi131.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.7 kB] 2024-06-15T07:24:09.390 INFO:teuthology.orchestra.run.smithi131.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-15T07:24:09.390 INFO:teuthology.orchestra.run.smithi131.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-15T07:24:09.391 INFO:teuthology.orchestra.run.smithi131.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-15T07:24:09.393 INFO:teuthology.orchestra.run.smithi131.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-15T07:24:09.394 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.7 kB] 2024-06-15T07:24:09.395 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-15T07:24:09.395 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-15T07:24:09.396 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-15T07:24:09.397 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-15T07:24:09.411 INFO:teuthology.orchestra.run.smithi131.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-15T07:24:09.411 INFO:teuthology.orchestra.run.smithi131.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-15T07:24:09.412 INFO:teuthology.orchestra.run.smithi131.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-06-15T07:24:09.428 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-15T07:24:09.476 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-15T07:24:09.476 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-15T07:24:09.477 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-06-15T07:24:09.477 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.3 kB] 2024-06-15T07:24:09.477 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-15T07:24:09.492 INFO:teuthology.orchestra.run.smithi131.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.3 kB] 2024-06-15T07:24:09.493 INFO:teuthology.orchestra.run.smithi131.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-15T07:24:14.233 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 14.0 MB in 2s (7,298 kB/s) 2024-06-15T07:24:15.183 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 14.0 MB in 3s (5,402 kB/s) 2024-06-15T07:24:15.628 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:24:15.647 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T07:24:15.703 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:24:15.886 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2024-06-15T07:24:15.887 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2024-06-15T07:24:16.045 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:24:16.045 INFO:teuthology.orchestra.run.smithi131.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:24:16.045 INFO:teuthology.orchestra.run.smithi131.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:24:16.046 INFO:teuthology.orchestra.run.smithi131.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:24:16.046 INFO:teuthology.orchestra.run.smithi131.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-15T07:24:16.046 INFO:teuthology.orchestra.run.smithi131.stdout: python2.7-minimal 2024-06-15T07:24:16.046 INFO:teuthology.orchestra.run.smithi131.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:24:16.047 INFO:teuthology.orchestra.run.smithi131.stdout:The following additional packages will be installed: 2024-06-15T07:24:16.047 INFO:teuthology.orchestra.run.smithi131.stdout: linux-generic linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T07:24:16.047 INFO:teuthology.orchestra.run.smithi131.stdout: linux-headers-generic linux-image-5.15.0-112-generic 2024-06-15T07:24:16.047 INFO:teuthology.orchestra.run.smithi131.stdout: linux-modules-5.15.0-112-generic linux-modules-extra-5.15.0-112-generic 2024-06-15T07:24:16.049 INFO:teuthology.orchestra.run.smithi131.stdout:Suggested packages: 2024-06-15T07:24:16.049 INFO:teuthology.orchestra.run.smithi131.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-15T07:24:16.049 INFO:teuthology.orchestra.run.smithi131.stdout:Recommended packages: 2024-06-15T07:24:16.049 INFO:teuthology.orchestra.run.smithi131.stdout: thermald 2024-06-15T07:24:16.086 INFO:teuthology.orchestra.run.smithi131.stdout:The following NEW packages will be installed: 2024-06-15T07:24:16.086 INFO:teuthology.orchestra.run.smithi131.stdout: linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T07:24:16.086 INFO:teuthology.orchestra.run.smithi131.stdout: linux-image-5.15.0-112-generic linux-modules-5.15.0-112-generic 2024-06-15T07:24:16.086 INFO:teuthology.orchestra.run.smithi131.stdout: linux-modules-extra-5.15.0-112-generic 2024-06-15T07:24:16.087 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages will be upgraded: 2024-06-15T07:24:16.088 INFO:teuthology.orchestra.run.smithi131.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-15T07:24:16.143 INFO:teuthology.orchestra.run.smithi131.stdout:3 upgraded, 5 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T07:24:16.144 INFO:teuthology.orchestra.run.smithi131.stdout:Need to get 113 MB of archives. 2024-06-15T07:24:16.144 INFO:teuthology.orchestra.run.smithi131.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-15T07:24:16.144 INFO:teuthology.orchestra.run.smithi131.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-112-generic amd64 5.15.0-112.122 [22.6 MB] 2024-06-15T07:24:16.665 INFO:teuthology.orchestra.run.smithi131.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-112-generic amd64 5.15.0-112.122 [11.5 MB] 2024-06-15T07:24:16.706 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:24:16.723 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T07:24:16.777 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:24:16.813 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-112-generic amd64 5.15.0-112.122 [63.9 MB] 2024-06-15T07:24:16.961 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-15T07:24:16.962 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-15T07:24:17.126 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:24:17.126 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:24:17.126 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:24:17.127 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:24:17.127 INFO:teuthology.orchestra.run.smithi012.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-15T07:24:17.127 INFO:teuthology.orchestra.run.smithi012.stdout: python2.7-minimal 2024-06-15T07:24:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:24:17.127 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-06-15T07:24:17.128 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T07:24:17.128 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-generic linux-image-5.15.0-112-generic 2024-06-15T07:24:17.128 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-5.15.0-112-generic linux-modules-extra-5.15.0-112-generic 2024-06-15T07:24:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-06-15T07:24:17.129 INFO:teuthology.orchestra.run.smithi012.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-15T07:24:17.129 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-06-15T07:24:17.129 INFO:teuthology.orchestra.run.smithi012.stdout: thermald 2024-06-15T07:24:17.179 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-06-15T07:24:17.179 INFO:teuthology.orchestra.run.smithi012.stdout: linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T07:24:17.179 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.15.0-112-generic linux-modules-5.15.0-112-generic 2024-06-15T07:24:17.180 INFO:teuthology.orchestra.run.smithi012.stdout: linux-modules-extra-5.15.0-112-generic 2024-06-15T07:24:17.181 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages will be upgraded: 2024-06-15T07:24:17.181 INFO:teuthology.orchestra.run.smithi012.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-15T07:24:17.361 INFO:teuthology.orchestra.run.smithi012.stdout:3 upgraded, 5 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T07:24:17.361 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 113 MB of archives. 2024-06-15T07:24:17.361 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-15T07:24:17.361 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-112-generic amd64 5.15.0-112.122 [22.6 MB] 2024-06-15T07:24:18.018 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.112.112 [1,692 B] 2024-06-15T07:24:18.019 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.112.112 [2,510 B] 2024-06-15T07:24:18.019 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112 all 5.15.0-112.122 [12.3 MB] 2024-06-15T07:24:18.190 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112-generic amd64 5.15.0-112.122 [2,904 kB] 2024-06-15T07:24:18.226 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.112.112 [2,360 B] 2024-06-15T07:24:18.637 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 113 MB in 2s (53.2 MB/s) 2024-06-15T07:24:18.798 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-112-generic amd64 5.15.0-112.122 [11.5 MB] 2024-06-15T07:24:18.813 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-modules-5.15.0-112-generic. 2024-06-15T07:24:19.208 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-112-generic amd64 5.15.0-112.122 [63.9 MB] 2024-06-15T07:24:20.781 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-15T07:24:20.785 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../0-linux-modules-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:20.844 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:21.390 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.112.112 [1,692 B] 2024-06-15T07:24:21.390 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.112.112 [2,510 B] 2024-06-15T07:24:21.390 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112 all 5.15.0-112.122 [12.3 MB] 2024-06-15T07:24:22.054 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112-generic amd64 5.15.0-112.122 [2,904 kB] 2024-06-15T07:24:22.169 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.112.112 [2,360 B] 2024-06-15T07:24:22.589 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 113 MB in 5s (22.8 MB/s) 2024-06-15T07:24:22.778 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-5.15.0-112-generic. 2024-06-15T07:24:22.929 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-image-5.15.0-112-generic. 2024-06-15T07:24:22.951 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../1-linux-image-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:23.037 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:23.507 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-112-generic. 2024-06-15T07:24:23.528 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:23.566 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:24.881 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-15T07:24:24.885 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../0-linux-modules-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:24.984 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:27.194 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-image-5.15.0-112-generic. 2024-06-15T07:24:27.216 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../1-linux-image-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:27.321 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:27.772 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-112-generic. 2024-06-15T07:24:27.787 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:27.830 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:30.335 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../3-linux-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:30.538 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:30.902 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:30.999 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-image-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:31.301 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-headers-5.15.0-112. 2024-06-15T07:24:31.312 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../5-linux-headers-5.15.0-112_5.15.0-112.122_all.deb ... 2024-06-15T07:24:31.352 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T07:24:34.700 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../3-linux-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:34.825 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:35.292 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:35.403 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-image-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:35.739 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-112. 2024-06-15T07:24:35.762 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../5-linux-headers-5.15.0-112_5.15.0-112.122_all.deb ... 2024-06-15T07:24:35.806 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T07:24:37.086 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-headers-5.15.0-112-generic. 2024-06-15T07:24:37.123 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../6-linux-headers-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:37.161 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:39.692 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:39.815 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:40.157 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T07:24:40.262 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:40.388 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-generic (5.15.0.112.112) ... 2024-06-15T07:24:40.515 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:41.402 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package linux-headers-5.15.0-112-generic. 2024-06-15T07:24:41.439 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../6-linux-headers-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T07:24:41.476 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:41.851 INFO:teuthology.orchestra.run.smithi131.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:24:41.851 INFO:teuthology.orchestra.run.smithi131.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-15T07:24:41.851 INFO:teuthology.orchestra.run.smithi131.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:24:41.851 INFO:teuthology.orchestra.run.smithi131.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-112-generic 2024-06-15T07:24:42.003 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:43.379 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:44.856 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-image-generic (5.15.0.112.112) ... 2024-06-15T07:24:44.981 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-generic (5.15.0.112.112) ... 2024-06-15T07:24:45.108 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:45.173 INFO:teuthology.orchestra.run.smithi131.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-15T07:24:45.173 INFO:teuthology.orchestra.run.smithi131.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:24:45.725 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T07:24:45.847 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking linux-headers-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T07:24:46.214 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T07:24:46.335 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:46.461 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-headers-generic (5.15.0.112.112) ... 2024-06-15T07:24:46.613 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:47.955 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:24:47.955 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-15T07:24:47.955 INFO:teuthology.orchestra.run.smithi012.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:24:47.955 INFO:teuthology.orchestra.run.smithi012.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-112-generic 2024-06-15T07:24:48.125 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:50.209 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:51.635 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-image-generic (5.15.0.112.112) ... 2024-06-15T07:24:51.761 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up linux-generic (5.15.0.112.112) ... 2024-06-15T07:24:51.887 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T07:24:51.956 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-15T07:24:51.956 INFO:teuthology.orchestra.run.smithi012.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:24:59.915 INFO:teuthology.orchestra.run.smithi131.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-15T07:24:59.916 INFO:teuthology.orchestra.run.smithi131.stdout:Sourcing file `/etc/default/grub' 2024-06-15T07:24:59.933 INFO:teuthology.orchestra.run.smithi131.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T07:24:59.951 INFO:teuthology.orchestra.run.smithi131.stdout:Generating grub configuration file ... 2024-06-15T07:25:00.220 INFO:teuthology.orchestra.run.smithi131.stdout:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:25:00.246 INFO:teuthology.orchestra.run.smithi131.stdout:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:25:00.525 INFO:teuthology.orchestra.run.smithi131.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:25:00.530 INFO:teuthology.orchestra.run.smithi131.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T07:25:00.601 INFO:teuthology.orchestra.run.smithi131.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T07:25:00.606 INFO:teuthology.orchestra.run.smithi131.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T07:25:00.784 INFO:teuthology.orchestra.run.smithi131.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T07:25:00.784 INFO:teuthology.orchestra.run.smithi131.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T07:25:00.784 INFO:teuthology.orchestra.run.smithi131.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T07:25:00.835 INFO:teuthology.orchestra.run.smithi131.stdout:done 2024-06-15T07:25:01.220 DEBUG:teuthology.orchestra.run.smithi131:> dpkg -s linux-image-generic 2024-06-15T07:25:01.242 INFO:teuthology.orchestra.run.smithi131.stdout:Package: linux-image-generic 2024-06-15T07:25:01.242 INFO:teuthology.orchestra.run.smithi131.stdout:Status: install ok installed 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Priority: optional 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Section: kernel 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Installed-Size: 21 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: amd64 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Source: linux-meta 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Version: 5.15.0.112.112 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Recommends: thermald 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout:Description: Generic Linux kernel image 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout: This package will always depend on the latest generic kernel image 2024-06-15T07:25:01.243 INFO:teuthology.orchestra.run.smithi131.stdout: available. 2024-06-15T07:25:01.244 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-112-generic 2024-06-15T07:25:01.244 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-15T07:25:01.244 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-15T07:25:01.244 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2024-06-15T07:25:01.244 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-06-15T07:25:01.244 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get clean 2024-06-15T07:25:01.400 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2024-06-15T07:25:01.648 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:25:01.649 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T07:25:01.715 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-15T07:25:01.797 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T07:25:03.060 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:25:03.078 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T07:25:03.131 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:25:03.312 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2024-06-15T07:25:03.312 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2024-06-15T07:25:03.472 INFO:teuthology.orchestra.run.smithi131.stdout:linux-image-generic is already the newest version (5.15.0.112.112). 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T07:25:03.473 INFO:teuthology.orchestra.run.smithi131.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:25:03.512 INFO:teuthology.orchestra.run.smithi131.stdout:0 upgraded, 0 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T07:25:03.515 DEBUG:teuthology.orchestra.run.smithi131:> dpkg -s linux-image-generic 2024-06-15T07:25:03.570 INFO:teuthology.orchestra.run.smithi131.stdout:Package: linux-image-generic 2024-06-15T07:25:03.570 INFO:teuthology.orchestra.run.smithi131.stdout:Status: install ok installed 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Priority: optional 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Section: kernel 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Installed-Size: 21 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: amd64 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Source: linux-meta 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Version: 5.15.0.112.112 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Recommends: thermald 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout:Description: Generic Linux kernel image 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout: This package will always depend on the latest generic kernel image 2024-06-15T07:25:03.571 INFO:teuthology.orchestra.run.smithi131.stdout: available. 2024-06-15T07:25:03.572 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-06-15T07:25:03.620 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.4ZC0X5iiaH 2024-06-15T07:25:03.620 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4ZC0X5iiaH 2024-06-15T07:25:03.676 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.4ZC0X5iiaH 2024-06-15T07:25:03.806 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.4ZC0X5iiaH is 10KB 2024-06-15T07:25:03.819 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.4ZC0X5iiaH 2024-06-15T07:25:03.825 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-15T07:25:03.880 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:25:03.880 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-15T07:25:03.880 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-15T07:25:03.947 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-112-generic 2024-06-15T07:25:03.947 DEBUG:teuthology.orchestra.run.smithi131:> sudo update-grub 2024-06-15T07:25:05.213 INFO:teuthology.orchestra.run.smithi131.stderr:Sourcing file `/etc/default/grub' 2024-06-15T07:25:05.214 INFO:teuthology.orchestra.run.smithi131.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T07:25:05.217 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-06-15T07:25:05.389 INFO:teuthology.orchestra.run.smithi131.stderr:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:25:05.395 INFO:teuthology.orchestra.run.smithi131.stderr:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:25:05.612 INFO:teuthology.orchestra.run.smithi131.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:25:05.617 INFO:teuthology.orchestra.run.smithi131.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T07:25:05.655 INFO:teuthology.orchestra.run.smithi131.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T07:25:05.661 INFO:teuthology.orchestra.run.smithi131.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T07:25:05.797 INFO:teuthology.orchestra.run.smithi131.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T07:25:05.797 INFO:teuthology.orchestra.run.smithi131.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T07:25:05.797 INFO:teuthology.orchestra.run.smithi131.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T07:25:05.819 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-06-15T07:25:05.829 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-06-15T07:25:06.631 INFO:teuthology.orchestra.run.smithi012.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-15T07:25:06.632 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub' 2024-06-15T07:25:06.661 INFO:teuthology.orchestra.run.smithi012.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T07:25:06.682 INFO:teuthology.orchestra.run.smithi012.stdout:Generating grub configuration file ... 2024-06-15T07:25:06.960 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:25:06.988 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:25:07.283 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:25:07.288 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T07:25:07.362 INFO:teuthology.orchestra.run.smithi012.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T07:25:07.367 INFO:teuthology.orchestra.run.smithi012.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T07:25:07.550 INFO:teuthology.orchestra.run.smithi012.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T07:25:07.551 INFO:teuthology.orchestra.run.smithi012.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T07:25:07.551 INFO:teuthology.orchestra.run.smithi012.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T07:25:07.590 INFO:teuthology.orchestra.run.smithi012.stdout:done 2024-06-15T07:25:08.086 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.112.112 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T07:25:08.109 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T07:25:08.110 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-06-15T07:25:08.110 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-06-15T07:25:08.110 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-06-15T07:25:08.110 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-06-15T07:25:08.111 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-112-generic 2024-06-15T07:25:08.111 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-15T07:25:08.111 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-15T07:25:08.111 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2024-06-15T07:25:08.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-06-15T07:25:08.111 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2024-06-15T07:25:08.282 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-15T07:25:08.410 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T07:25:08.524 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:25:08.625 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-15T07:25:08.687 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T07:25:09.853 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:25:09.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T07:25:09.935 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:25:10.139 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-15T07:25:10.139 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-15T07:25:10.323 INFO:teuthology.orchestra.run.smithi012.stdout:linux-image-generic is already the newest version (5.15.0.112.112). 2024-06-15T07:25:10.323 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:25:10.323 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:25:10.323 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:25:10.323 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:25:10.324 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T07:25:10.324 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T07:25:10.324 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:25:10.369 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 0 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T07:25:10.371 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2024-06-15T07:25:10.387 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2024-06-15T07:25:10.387 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 21 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 5.15.0.112.112 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2024-06-15T07:25:10.388 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2024-06-15T07:25:10.389 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-06-15T07:25:10.435 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.YcznIYi0nL 2024-06-15T07:25:10.436 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub/grub.cfg /tmp/tmp.YcznIYi0nL 2024-06-15T07:25:10.491 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.YcznIYi0nL 2024-06-15T07:25:10.608 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.YcznIYi0nL is 10KB 2024-06-15T07:25:10.621 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.YcznIYi0nL 2024-06-15T07:25:10.626 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-15T07:25:10.683 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:25:10.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-15T07:25:10.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-15T07:25:10.753 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-112-generic 2024-06-15T07:25:10.753 DEBUG:teuthology.orchestra.run.smithi012:> sudo update-grub 2024-06-15T07:25:12.094 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub' 2024-06-15T07:25:12.095 INFO:teuthology.orchestra.run.smithi012.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T07:25:12.098 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-06-15T07:25:12.288 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T07:25:12.294 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T07:25:12.550 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T07:25:12.555 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T07:25:12.601 INFO:teuthology.orchestra.run.smithi012.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T07:25:12.607 INFO:teuthology.orchestra.run.smithi012.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T07:25:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T07:25:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T07:25:12.755 INFO:teuthology.orchestra.run.smithi012.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T07:25:12.766 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-06-15T07:25:12.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-06-15T07:25:35.854 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2024-06-15T07:25:35.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-06-15T07:25:35.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:25:42.772 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2024-06-15T07:25:42.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-15T07:25:42.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:26:35.905 DEBUG:teuthology.orchestra.remote:timed out 2024-06-15T07:26:42.781 DEBUG:teuthology.orchestra.remote:timed out 2024-06-15T07:26:44.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-06-15T07:26:44.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:26:45.490 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-06-15T07:26:46.267 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-06-15T07:26:46.267 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.15.0-112-generic"... 2024-06-15T07:26:46.267 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-06-15T07:26:46.317 INFO:teuthology.orchestra.run.smithi131.stdout:5.15.0-112-generic 2024-06-15T07:26:46.318 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-112-generic vs 5.15.0-112-generic 2024-06-15T07:26:46.318 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-15T07:26:46.318 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-15T07:26:47.319 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-06-15T07:26:47.319 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-15T07:26:47.405 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-06-15T07:26:47.423 DEBUG:teuthology.parallel:result is None 2024-06-15T07:26:51.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-15T07:26:51.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:26:51.982 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-06-15T07:26:52.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-15T07:26:52.565 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.15.0-112-generic"... 2024-06-15T07:26:52.565 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-06-15T07:26:52.613 INFO:teuthology.orchestra.run.smithi012.stdout:5.15.0-112-generic 2024-06-15T07:26:52.613 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-112-generic vs 5.15.0-112-generic 2024-06-15T07:26:52.613 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-15T07:26:52.613 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-15T07:26:53.614 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-06-15T07:26:53.614 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-15T07:26:53.685 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-06-15T07:26:53.702 DEBUG:teuthology.parallel:result is None 2024-06-15T07:26:53.702 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-15T07:26:53.709 INFO:teuthology.task.internal:Creating test directory... 2024-06-15T07:26:53.709 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-15T07:26:53.711 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-15T07:26:53.717 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-15T07:26:53.743 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-15T07:26:53.797 INFO:teuthology.task.internal:Creating archive directory... 2024-06-15T07:26:53.798 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-15T07:26:53.800 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-15T07:26:53.882 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-15T07:26:53.888 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-15T07:26:53.888 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-15T07:26:53.891 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-15T07:26:53.910 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T07:26:53.911 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T07:26:53.918 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T07:26:53.919 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T07:26:53.920 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-15T07:26:53.928 INFO:teuthology.task.internal:Configuring sudo... 2024-06-15T07:26:53.928 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-15T07:26:53.966 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-15T07:26:53.979 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-15T07:26:53.987 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-15T07:26:53.988 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-15T07:26:54.022 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-15T07:26:54.027 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T07:26:54.073 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T07:26:54.125 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:26:54.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-15T07:26:54.183 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T07:26:54.189 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T07:26:54.238 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:26:54.238 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-15T07:26:54.302 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-06-15T07:26:54.305 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-06-15T07:26:54.372 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-15T07:26:54.380 INFO:teuthology.task.internal:Starting timer... 2024-06-15T07:26:54.380 INFO:teuthology.run_tasks:Running task pcp... 2024-06-15T07:26:54.424 INFO:teuthology.run_tasks:Running task selinux... 2024-06-15T07:26:54.455 DEBUG:teuthology.task.selinux:Excluding smithi012: OS 'ubuntu' does not support SELinux 2024-06-15T07:26:54.455 DEBUG:teuthology.task.selinux:Excluding smithi131: OS 'ubuntu' does not support SELinux 2024-06-15T07:26:54.455 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-15T07:26:54.455 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-06-15T07:26:54.455 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-15T07:26:54.456 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-15T07:26:54.488 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-15T07:26:54.492 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-15T07:26:54.601 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-15T07:26:54.612 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-15T07:26:54.613 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-15T07:33:54.162 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2024-06-15T07:33:54.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-15T07:33:54.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:33:54.247 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-06-15T07:33:54.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-06-15T07:33:54.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-06-15T07:33:54.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T07:33:54.402 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-06-15T07:33:54.482 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-06-15T07:33:54.483 INFO:teuthology.run_tasks:Running task clock... 2024-06-15T07:33:54.493 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-15T07:33:54.493 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-15T07:33:54.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T07:33:54.496 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-15T07:33:54.496 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Command line: ntpd -gq 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: ---------------------------------------------------- 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: ntp-4 is maintained by Network Time Foundation, 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: corporation. Support and training for ntp-4 are 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: available at https://www.nwtime.org/support 2024-06-15T07:33:54.525 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: ---------------------------------------------------- 2024-06-15T07:33:54.526 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: proto: precision = 0.063 usec (-24) 2024-06-15T07:33:54.526 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: basedate set to 2022-02-04 2024-06-15T07:33:54.526 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: gps base set to 2022-02-06 (week 2196) 2024-06-15T07:33:54.526 INFO:teuthology.orchestra.run.smithi012.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T07:33:54.527 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T07:33:54.527 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: restrict ::: KOD does nothing without LIMITED. 2024-06-15T07:33:54.527 INFO:teuthology.orchestra.run.smithi012.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen and drop on 0 v6wildcard [::]:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen normally on 2 lo 127.0.0.1:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen normally on 3 ens1f0 172.21.15.12:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen normally on 4 lo [::1]:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25ea%4]:123 2024-06-15T07:33:54.528 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:33:54 ntpd[17492]: Listening on routing socket on fd #22 for interface updates 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Command line: ntpd -gq 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: ---------------------------------------------------- 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: ntp-4 is maintained by Network Time Foundation, 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: corporation. Support and training for ntp-4 are 2024-06-15T07:33:54.549 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: available at https://www.nwtime.org/support 2024-06-15T07:33:54.550 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: ---------------------------------------------------- 2024-06-15T07:33:54.550 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: proto: precision = 0.056 usec (-24) 2024-06-15T07:33:54.550 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: basedate set to 2022-02-04 2024-06-15T07:33:54.550 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: gps base set to 2022-02-06 (week 2196) 2024-06-15T07:33:54.550 INFO:teuthology.orchestra.run.smithi131.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T07:33:54.551 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T07:33:54.551 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: restrict ::: KOD does nothing without LIMITED. 2024-06-15T07:33:54.551 INFO:teuthology.orchestra.run.smithi131.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen and drop on 0 v6wildcard [::]:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen normally on 2 lo 127.0.0.1:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen normally on 3 enp3s0f1 172.21.15.131:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen normally on 4 lo [::1]:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:72f9%5]:123 2024-06-15T07:33:54.552 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:33:54 ntpd[17563]: Listening on routing socket on fd #22 for interface updates 2024-06-15T07:33:55.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:55.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:55.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:55.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:55.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:55.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:55.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:55.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:55 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:56.528 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:56.529 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:56 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:56.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:56 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:57.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:57 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:58.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:58 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:58.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:58.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:58 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:59.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:59.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:33:59 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:33:59.549 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:59.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:59.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:33:59.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:33:59.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:33:59.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:33:59 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:00.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:00.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:00 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:00.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:00.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:00 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:01.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:01 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:01.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:01.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:01.550 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:01.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:01.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:01.551 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:01.551 INFO:teuthology.orchestra.run.smithi131.stdout:15 Jun 07:34:01 ntpd[17563]: ntpd: time slew -0.001403 s 2024-06-15T07:34:01.551 INFO:teuthology.orchestra.run.smithi131.stdout:ntpd: time slew -0.001403s 2024-06-15T07:34:01.552 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-15T07:34:01.552 INFO:teuthology.orchestra.run.smithi131.stderr:15 Jun 07:34:01 ntpd[17563]: can't open /var/log/ntpstats/loopstats.20240615: Permission denied 2024-06-15T07:34:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T07:34:01.598 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================== 2024-06-15T07:34:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T07:34:02.526 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T07:34:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T07:34:02.527 INFO:teuthology.orchestra.run.smithi012.stdout:15 Jun 07:34:02 ntpd[17492]: ntpd: time slew -0.000018 s 2024-06-15T07:34:02.527 INFO:teuthology.orchestra.run.smithi012.stdout:ntpd: time slew -0.000018s 2024-06-15T07:34:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-15T07:34:02.527 INFO:teuthology.orchestra.run.smithi012.stderr:15 Jun 07:34:02 ntpd[17492]: can't open /var/log/ntpstats/loopstats.20240615: Permission denied 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:02.578 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T07:34:02.579 INFO:teuthology.run_tasks:Running task install... 2024-06-15T07:34:02.589 DEBUG:teuthology.task.install:project ceph 2024-06-15T07:34:02.590 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}} 2024-06-15T07:34:02.590 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-15T07:34:02.590 INFO:teuthology.task.install:Using flavor: default 2024-06-15T07:34:02.595 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-15T07:34:02.596 INFO:teuthology.task.install:extra packages: [] 2024-06-15T07:34:02.596 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-key list | grep Ceph 2024-06-15T07:34:02.597 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-key list | grep Ceph 2024-06-15T07:34:02.659 INFO:teuthology.orchestra.run.smithi131.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-15T07:34:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-15T07:34:02.688 INFO:teuthology.orchestra.run.smithi131.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-15T07:34:02.689 INFO:teuthology.orchestra.run.smithi131.stdout:uid [ unknown] Ceph.com (release key) 2024-06-15T07:34:02.690 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-15T07:34:02.690 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T07:34:02.713 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-15T07:34:02.713 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph.com (release key) 2024-06-15T07:34:02.714 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-15T07:34:02.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T07:34:02.854 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default/ 2024-06-15T07:34:02.854 INFO:teuthology.task.install.deb:Package version is 18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:34:02.909 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default/ 2024-06-15T07:34:02.909 INFO:teuthology.task.install.deb:Package version is 18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:34:02.964 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:34:02.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-15T07:34:02.979 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2024-06-15T07:34:03.089 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:34:03.089 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-15T07:34:03.103 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2024-06-15T07:34:03.153 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:34:03.166 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-15T07:34:03.187 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T07:34:03.260 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy InRelease 2024-06-15T07:34:03.268 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T07:34:03.296 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-06-15T07:34:03.334 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-15T07:34:03.370 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,430 B] 2024-06-15T07:34:03.387 INFO:teuthology.orchestra.run.smithi131.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy InRelease 2024-06-15T07:34:03.403 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T07:34:03.403 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T07:34:03.411 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-06-15T07:34:03.423 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-06-15T07:34:03.461 INFO:teuthology.orchestra.run.smithi131.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-15T07:34:03.474 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-15T07:34:03.498 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,430 B] 2024-06-15T07:34:03.520 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 28.8 kB in 0s (79.7 kB/s) 2024-06-15T07:34:03.541 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2024-06-15T07:34:03.556 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T07:34:03.791 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 28.8 kB in 1s (57.2 kB/s) 2024-06-15T07:34:04.528 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:34:04.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1443-g6f4b0956-1jammy cephadm=18.2.2-1443-g6f4b0956-1jammy ceph-mds=18.2.2-1443-g6f4b0956-1jammy ceph-mgr=18.2.2-1443-g6f4b0956-1jammy ceph-common=18.2.2-1443-g6f4b0956-1jammy ceph-fuse=18.2.2-1443-g6f4b0956-1jammy ceph-test=18.2.2-1443-g6f4b0956-1jammy ceph-volume=18.2.2-1443-g6f4b0956-1jammy radosgw=18.2.2-1443-g6f4b0956-1jammy python3-rados=18.2.2-1443-g6f4b0956-1jammy python3-rgw=18.2.2-1443-g6f4b0956-1jammy python3-cephfs=18.2.2-1443-g6f4b0956-1jammy python3-rbd=18.2.2-1443-g6f4b0956-1jammy libcephfs2=18.2.2-1443-g6f4b0956-1jammy libcephfs-dev=18.2.2-1443-g6f4b0956-1jammy librados2=18.2.2-1443-g6f4b0956-1jammy librbd1=18.2.2-1443-g6f4b0956-1jammy rbd-fuse=18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:34:04.600 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2024-06-15T07:34:04.779 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:34:04.783 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2024-06-15T07:34:04.784 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2024-06-15T07:34:04.795 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.2-1443-g6f4b0956-1jammy cephadm=18.2.2-1443-g6f4b0956-1jammy ceph-mds=18.2.2-1443-g6f4b0956-1jammy ceph-mgr=18.2.2-1443-g6f4b0956-1jammy ceph-common=18.2.2-1443-g6f4b0956-1jammy ceph-fuse=18.2.2-1443-g6f4b0956-1jammy ceph-test=18.2.2-1443-g6f4b0956-1jammy ceph-volume=18.2.2-1443-g6f4b0956-1jammy radosgw=18.2.2-1443-g6f4b0956-1jammy python3-rados=18.2.2-1443-g6f4b0956-1jammy python3-rgw=18.2.2-1443-g6f4b0956-1jammy python3-cephfs=18.2.2-1443-g6f4b0956-1jammy python3-rbd=18.2.2-1443-g6f4b0956-1jammy libcephfs2=18.2.2-1443-g6f4b0956-1jammy libcephfs-dev=18.2.2-1443-g6f4b0956-1jammy librados2=18.2.2-1443-g6f4b0956-1jammy librbd1=18.2.2-1443-g6f4b0956-1jammy rbd-fuse=18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:34:04.858 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2024-06-15T07:34:04.966 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:34:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:34:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:34:04.966 INFO:teuthology.orchestra.run.smithi012.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:34:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T07:34:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T07:34:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:34:04.968 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2024-06-15T07:34:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-15T07:34:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-15T07:34:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-06-15T07:34:04.968 INFO:teuthology.orchestra.run.smithi012.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-15T07:34:04.969 INFO:teuthology.orchestra.run.smithi012.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-06-15T07:34:04.970 INFO:teuthology.orchestra.run.smithi012.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: python3-watchdog 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout:Recommended packages: 2024-06-15T07:34:04.971 INFO:teuthology.orchestra.run.smithi012.stdout: btrfs-tools 2024-06-15T07:34:05.060 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2024-06-15T07:34:05.060 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2024-06-15T07:34:05.073 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2024-06-15T07:34:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-15T07:34:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-15T07:34:05.073 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-15T07:34:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-06-15T07:34:05.074 INFO:teuthology.orchestra.run.smithi012.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-06-15T07:34:05.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-06-15T07:34:05.076 INFO:teuthology.orchestra.run.smithi012.stdout: socat xmlstarlet 2024-06-15T07:34:05.140 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 109 newly installed, 0 to remove and 313 not upgraded. 2024-06-15T07:34:05.141 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 162 MB of archives. 2024-06-15T07:34:05.141 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 744 MB of additional disk space will be used. 2024-06-15T07:34:05.141 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-06-15T07:34:05.211 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T07:34:05.252 INFO:teuthology.orchestra.run.smithi131.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T07:34:05.253 INFO:teuthology.orchestra.run.smithi131.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T07:34:05.253 INFO:teuthology.orchestra.run.smithi131.stdout:The following additional packages will be installed: 2024-06-15T07:34:05.253 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-15T07:34:05.254 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-15T07:34:05.254 INFO:teuthology.orchestra.run.smithi131.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-06-15T07:34:05.254 INFO:teuthology.orchestra.run.smithi131.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-15T07:34:05.254 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-15T07:34:05.255 INFO:teuthology.orchestra.run.smithi131.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout:Suggested packages: 2024-06-15T07:34:05.256 INFO:teuthology.orchestra.run.smithi131.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: python3-watchdog 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout:Recommended packages: 2024-06-15T07:34:05.257 INFO:teuthology.orchestra.run.smithi131.stdout: btrfs-tools 2024-06-15T07:34:05.258 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-15T07:34:05.276 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-15T07:34:05.294 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1443-g6f4b0956-1jammy [3,448 kB] 2024-06-15T07:34:05.337 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-15T07:34:05.342 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout:The following NEW packages will be installed: 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-06-15T07:34:05.355 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-06-15T07:34:05.356 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-06-15T07:34:05.357 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-06-15T07:34:05.358 INFO:teuthology.orchestra.run.smithi131.stdout: socat xmlstarlet 2024-06-15T07:34:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-15T07:34:05.359 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-15T07:34:05.360 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-15T07:34:05.369 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-15T07:34:05.370 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-15T07:34:05.370 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-15T07:34:05.372 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-15T07:34:05.383 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-15T07:34:05.383 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-15T07:34:05.383 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-15T07:34:05.403 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-15T07:34:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-15T07:34:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-15T07:34:05.407 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-15T07:34:05.408 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-15T07:34:05.408 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-15T07:34:05.409 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-15T07:34:05.420 INFO:teuthology.orchestra.run.smithi131.stdout:0 upgraded, 109 newly installed, 0 to remove and 313 not upgraded. 2024-06-15T07:34:05.420 INFO:teuthology.orchestra.run.smithi131.stdout:Need to get 162 MB of archives. 2024-06-15T07:34:05.420 INFO:teuthology.orchestra.run.smithi131.stdout:After this operation, 744 MB of additional disk space will be used. 2024-06-15T07:34:05.420 INFO:teuthology.orchestra.run.smithi131.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-06-15T07:34:05.424 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-15T07:34:05.424 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-15T07:34:05.425 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-15T07:34:05.445 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-15T07:34:05.446 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-15T07:34:05.446 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-15T07:34:05.452 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-15T07:34:05.452 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-15T07:34:05.453 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-15T07:34:05.454 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-15T07:34:05.465 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-15T07:34:05.468 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-15T07:34:05.468 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-15T07:34:05.486 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-15T07:34:05.487 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-15T07:34:05.490 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-15T07:34:05.491 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-15T07:34:05.491 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-15T07:34:05.494 INFO:teuthology.orchestra.run.smithi012.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-15T07:34:05.499 INFO:teuthology.orchestra.run.smithi131.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-15T07:34:05.551 INFO:teuthology.orchestra.run.smithi012.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-15T07:34:05.552 INFO:teuthology.orchestra.run.smithi012.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-15T07:34:05.553 INFO:teuthology.orchestra.run.smithi012.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-15T07:34:05.553 INFO:teuthology.orchestra.run.smithi012.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-15T07:34:05.554 INFO:teuthology.orchestra.run.smithi012.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-15T07:34:05.554 INFO:teuthology.orchestra.run.smithi012.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-15T07:34:05.555 INFO:teuthology.orchestra.run.smithi012.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-15T07:34:05.555 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-15T07:34:05.558 INFO:teuthology.orchestra.run.smithi012.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-15T07:34:05.558 INFO:teuthology.orchestra.run.smithi012.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-15T07:34:05.566 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-15T07:34:05.573 INFO:teuthology.orchestra.run.smithi012.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-15T07:34:05.593 INFO:teuthology.orchestra.run.smithi012.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-15T07:34:05.595 INFO:teuthology.orchestra.run.smithi012.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-15T07:34:05.595 INFO:teuthology.orchestra.run.smithi012.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-15T07:34:05.604 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.2-1443-g6f4b0956-1jammy [3,448 kB] 2024-06-15T07:34:05.604 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-15T07:34:05.616 INFO:teuthology.orchestra.run.smithi012.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-15T07:34:05.616 INFO:teuthology.orchestra.run.smithi012.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-15T07:34:05.617 INFO:teuthology.orchestra.run.smithi012.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1443-g6f4b0956-1jammy [3,105 kB] 2024-06-15T07:34:05.620 INFO:teuthology.orchestra.run.smithi012.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-15T07:34:05.621 INFO:teuthology.orchestra.run.smithi012.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-15T07:34:05.621 INFO:teuthology.orchestra.run.smithi012.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-15T07:34:05.622 INFO:teuthology.orchestra.run.smithi012.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-15T07:34:05.627 INFO:teuthology.orchestra.run.smithi012.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-15T07:34:05.628 INFO:teuthology.orchestra.run.smithi012.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-15T07:34:05.632 INFO:teuthology.orchestra.run.smithi012.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-15T07:34:05.648 INFO:teuthology.orchestra.run.smithi012.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-15T07:34:05.649 INFO:teuthology.orchestra.run.smithi012.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-15T07:34:05.650 INFO:teuthology.orchestra.run.smithi012.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-15T07:34:05.653 INFO:teuthology.orchestra.run.smithi012.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-15T07:34:05.656 INFO:teuthology.orchestra.run.smithi012.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-15T07:34:05.660 INFO:teuthology.orchestra.run.smithi012.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-15T07:34:05.663 INFO:teuthology.orchestra.run.smithi012.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-15T07:34:05.677 INFO:teuthology.orchestra.run.smithi012.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-15T07:34:05.679 INFO:teuthology.orchestra.run.smithi012.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-15T07:34:05.690 INFO:teuthology.orchestra.run.smithi012.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-15T07:34:05.694 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-15T07:34:05.699 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-15T07:34:05.717 INFO:teuthology.orchestra.run.smithi131.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-15T07:34:05.719 INFO:teuthology.orchestra.run.smithi012.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-15T07:34:05.720 INFO:teuthology.orchestra.run.smithi012.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-15T07:34:05.728 INFO:teuthology.orchestra.run.smithi131.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-15T07:34:05.729 INFO:teuthology.orchestra.run.smithi131.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-15T07:34:05.731 INFO:teuthology.orchestra.run.smithi131.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-15T07:34:05.808 INFO:teuthology.orchestra.run.smithi131.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-15T07:34:05.809 INFO:teuthology.orchestra.run.smithi131.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-15T07:34:05.810 INFO:teuthology.orchestra.run.smithi131.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-15T07:34:05.812 INFO:teuthology.orchestra.run.smithi131.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-15T07:34:05.812 INFO:teuthology.orchestra.run.smithi131.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-15T07:34:05.813 INFO:teuthology.orchestra.run.smithi131.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-15T07:34:05.814 INFO:teuthology.orchestra.run.smithi131.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-15T07:34:05.830 INFO:teuthology.orchestra.run.smithi012.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1443-g6f4b0956-1jammy [680 kB] 2024-06-15T07:34:05.836 INFO:teuthology.orchestra.run.smithi131.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-15T07:34:05.837 INFO:teuthology.orchestra.run.smithi131.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-15T07:34:05.839 INFO:teuthology.orchestra.run.smithi131.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-15T07:34:05.847 INFO:teuthology.orchestra.run.smithi012.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1443-g6f4b0956-1jammy [357 kB] 2024-06-15T07:34:05.854 INFO:teuthology.orchestra.run.smithi012.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1443-g6f4b0956-1jammy [32.8 kB] 2024-06-15T07:34:05.855 INFO:teuthology.orchestra.run.smithi012.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1443-g6f4b0956-1jammy [180 kB] 2024-06-15T07:34:05.859 INFO:teuthology.orchestra.run.smithi012.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1443-g6f4b0956-1jammy [63.8 kB] 2024-06-15T07:34:05.861 INFO:teuthology.orchestra.run.smithi131.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-15T07:34:05.861 INFO:teuthology.orchestra.run.smithi012.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1443-g6f4b0956-1jammy [334 kB] 2024-06-15T07:34:05.862 INFO:teuthology.orchestra.run.smithi131.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-15T07:34:05.863 INFO:teuthology.orchestra.run.smithi131.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-15T07:34:05.864 INFO:teuthology.orchestra.run.smithi131.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-15T07:34:05.866 INFO:teuthology.orchestra.run.smithi131.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-15T07:34:05.866 INFO:teuthology.orchestra.run.smithi131.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-15T07:34:05.867 INFO:teuthology.orchestra.run.smithi131.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-15T07:34:05.868 INFO:teuthology.orchestra.run.smithi012.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1443-g6f4b0956-1jammy [5,891 kB] 2024-06-15T07:34:05.870 INFO:teuthology.orchestra.run.smithi131.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-15T07:34:05.871 INFO:teuthology.orchestra.run.smithi131.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-15T07:34:05.885 INFO:teuthology.orchestra.run.smithi131.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-15T07:34:05.897 INFO:teuthology.orchestra.run.smithi131.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-15T07:34:05.897 INFO:teuthology.orchestra.run.smithi131.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-15T07:34:05.899 INFO:teuthology.orchestra.run.smithi131.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-15T07:34:05.900 INFO:teuthology.orchestra.run.smithi131.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-15T07:34:05.902 INFO:teuthology.orchestra.run.smithi131.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-15T07:34:05.907 INFO:teuthology.orchestra.run.smithi131.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-15T07:34:05.908 INFO:teuthology.orchestra.run.smithi131.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-15T07:34:05.911 INFO:teuthology.orchestra.run.smithi131.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-15T07:34:05.913 INFO:teuthology.orchestra.run.smithi131.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-15T07:34:05.939 INFO:teuthology.orchestra.run.smithi131.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-15T07:34:05.940 INFO:teuthology.orchestra.run.smithi131.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-15T07:34:05.942 INFO:teuthology.orchestra.run.smithi131.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-15T07:34:05.947 INFO:teuthology.orchestra.run.smithi131.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-15T07:34:06.030 INFO:teuthology.orchestra.run.smithi131.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-15T07:34:06.031 INFO:teuthology.orchestra.run.smithi131.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-15T07:34:06.033 INFO:teuthology.orchestra.run.smithi131.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-15T07:34:06.034 INFO:teuthology.orchestra.run.smithi131.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-15T07:34:06.035 INFO:teuthology.orchestra.run.smithi131.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-15T07:34:06.035 INFO:teuthology.orchestra.run.smithi131.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-15T07:34:06.036 INFO:teuthology.orchestra.run.smithi131.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-15T07:34:06.039 INFO:teuthology.orchestra.run.smithi131.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-15T07:34:06.039 INFO:teuthology.orchestra.run.smithi131.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-15T07:34:06.054 INFO:teuthology.orchestra.run.smithi131.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-15T07:34:06.082 INFO:teuthology.orchestra.run.smithi131.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-15T07:34:06.085 INFO:teuthology.orchestra.run.smithi131.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-15T07:34:06.085 INFO:teuthology.orchestra.run.smithi131.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-15T07:34:06.114 INFO:teuthology.orchestra.run.smithi131.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-15T07:34:06.114 INFO:teuthology.orchestra.run.smithi131.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-15T07:34:06.116 INFO:teuthology.orchestra.run.smithi131.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-15T07:34:06.117 INFO:teuthology.orchestra.run.smithi131.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-15T07:34:06.118 INFO:teuthology.orchestra.run.smithi131.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-15T07:34:06.119 INFO:teuthology.orchestra.run.smithi131.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-15T07:34:06.123 INFO:teuthology.orchestra.run.smithi131.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-15T07:34:06.124 INFO:teuthology.orchestra.run.smithi131.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-15T07:34:06.129 INFO:teuthology.orchestra.run.smithi131.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-15T07:34:06.144 INFO:teuthology.orchestra.run.smithi131.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-15T07:34:06.147 INFO:teuthology.orchestra.run.smithi131.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-15T07:34:06.147 INFO:teuthology.orchestra.run.smithi131.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-15T07:34:06.152 INFO:teuthology.orchestra.run.smithi131.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-15T07:34:06.156 INFO:teuthology.orchestra.run.smithi131.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-15T07:34:06.162 INFO:teuthology.orchestra.run.smithi131.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-15T07:34:06.166 INFO:teuthology.orchestra.run.smithi131.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-15T07:34:06.169 INFO:teuthology.orchestra.run.smithi131.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-15T07:34:06.171 INFO:teuthology.orchestra.run.smithi131.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-15T07:34:06.181 INFO:teuthology.orchestra.run.smithi131.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-15T07:34:06.183 INFO:teuthology.orchestra.run.smithi012.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1443-g6f4b0956-1jammy [112 kB] 2024-06-15T07:34:06.185 INFO:teuthology.orchestra.run.smithi012.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1443-g6f4b0956-1jammy [441 kB] 2024-06-15T07:34:06.193 INFO:teuthology.orchestra.run.smithi131.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-15T07:34:06.194 INFO:teuthology.orchestra.run.smithi131.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-15T07:34:06.195 INFO:teuthology.orchestra.run.smithi012.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1443-g6f4b0956-1jammy [21.9 MB] 2024-06-15T07:34:07.008 INFO:teuthology.orchestra.run.smithi131.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.2-1443-g6f4b0956-1jammy [3,105 kB] 2024-06-15T07:34:07.199 INFO:teuthology.orchestra.run.smithi012.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1443-g6f4b0956-1jammy [5,002 kB] 2024-06-15T07:34:07.364 INFO:teuthology.orchestra.run.smithi012.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1443-g6f4b0956-1jammy [241 kB] 2024-06-15T07:34:07.370 INFO:teuthology.orchestra.run.smithi012.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1443-g6f4b0956-1jammy [130 kB] 2024-06-15T07:34:07.374 INFO:teuthology.orchestra.run.smithi012.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1443-g6f4b0956-1jammy [1,490 kB] 2024-06-15T07:34:07.427 INFO:teuthology.orchestra.run.smithi012.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1443-g6f4b0956-1jammy [6,080 kB] 2024-06-15T07:34:07.654 INFO:teuthology.orchestra.run.smithi012.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1443-g6f4b0956-1jammy [22.3 MB] 2024-06-15T07:34:07.917 INFO:teuthology.orchestra.run.smithi131.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.2-1443-g6f4b0956-1jammy [680 kB] 2024-06-15T07:34:08.112 INFO:teuthology.orchestra.run.smithi131.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.2-1443-g6f4b0956-1jammy [357 kB] 2024-06-15T07:34:08.229 INFO:teuthology.orchestra.run.smithi131.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.2-1443-g6f4b0956-1jammy [32.8 kB] 2024-06-15T07:34:08.230 INFO:teuthology.orchestra.run.smithi131.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.2-1443-g6f4b0956-1jammy [180 kB] 2024-06-15T07:34:08.301 INFO:teuthology.orchestra.run.smithi131.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.2-1443-g6f4b0956-1jammy [63.8 kB] 2024-06-15T07:34:08.308 INFO:teuthology.orchestra.run.smithi131.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.2-1443-g6f4b0956-1jammy [334 kB] 2024-06-15T07:34:08.391 INFO:teuthology.orchestra.run.smithi131.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.2-1443-g6f4b0956-1jammy [5,891 kB] 2024-06-15T07:34:08.566 INFO:teuthology.orchestra.run.smithi012.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1443-g6f4b0956-1jammy [14.1 kB] 2024-06-15T07:34:08.567 INFO:teuthology.orchestra.run.smithi012.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1443-g6f4b0956-1jammy [831 kB] 2024-06-15T07:34:08.583 INFO:teuthology.orchestra.run.smithi012.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1443-g6f4b0956-1jammy [2,154 kB] 2024-06-15T07:34:08.621 INFO:teuthology.orchestra.run.smithi012.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1443-g6f4b0956-1jammy [238 kB] 2024-06-15T07:34:08.629 INFO:teuthology.orchestra.run.smithi012.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1443-g6f4b0956-1jammy [145 kB] 2024-06-15T07:34:08.640 INFO:teuthology.orchestra.run.smithi012.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1443-g6f4b0956-1jammy [2,227 kB] 2024-06-15T07:34:08.719 INFO:teuthology.orchestra.run.smithi012.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1443-g6f4b0956-1jammy [8,625 kB] 2024-06-15T07:34:09.041 INFO:teuthology.orchestra.run.smithi012.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1443-g6f4b0956-1jammy [14.3 kB] 2024-06-15T07:34:09.041 INFO:teuthology.orchestra.run.smithi012.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1443-g6f4b0956-1jammy [42.6 MB] 2024-06-15T07:34:10.166 INFO:teuthology.orchestra.run.smithi131.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.2-1443-g6f4b0956-1jammy [112 kB] 2024-06-15T07:34:10.170 INFO:teuthology.orchestra.run.smithi131.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.2-1443-g6f4b0956-1jammy [441 kB] 2024-06-15T07:34:10.310 INFO:teuthology.orchestra.run.smithi131.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.2-1443-g6f4b0956-1jammy [21.9 MB] 2024-06-15T07:34:11.051 INFO:teuthology.orchestra.run.smithi012.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1443-g6f4b0956-1jammy [121 kB] 2024-06-15T07:34:11.054 INFO:teuthology.orchestra.run.smithi012.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1443-g6f4b0956-1jammy [38.3 kB] 2024-06-15T07:34:11.055 INFO:teuthology.orchestra.run.smithi012.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1443-g6f4b0956-1jammy [11.6 MB] 2024-06-15T07:34:11.446 INFO:teuthology.orchestra.run.smithi012.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1443-g6f4b0956-1jammy [93.1 kB] 2024-06-15T07:34:12.036 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 162 MB in 6s (25.5 MB/s) 2024-06-15T07:34:12.190 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libfmt8:amd64. 2024-06-15T07:34:12.293 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165982 files and directories currently installed.) 2024-06-15T07:34:12.299 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-06-15T07:34:12.380 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-06-15T07:34:12.716 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-15T07:34:12.743 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-15T07:34:12.783 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T07:34:13.220 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-15T07:34:13.247 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-15T07:34:13.287 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T07:34:13.606 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-15T07:34:13.633 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-15T07:34:13.673 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T07:34:14.118 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-15T07:34:14.145 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:14.193 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:14.630 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-15T07:34:14.657 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:14.696 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:14.931 INFO:teuthology.orchestra.run.smithi131.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.2-1443-g6f4b0956-1jammy [5,002 kB] 2024-06-15T07:34:14.999 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-15T07:34:15.027 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:15.057 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:15.403 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-15T07:34:15.430 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-15T07:34:15.469 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T07:34:15.697 INFO:teuthology.orchestra.run.smithi131.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.2-1443-g6f4b0956-1jammy [241 kB] 2024-06-15T07:34:15.734 INFO:teuthology.orchestra.run.smithi131.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.2-1443-g6f4b0956-1jammy [130 kB] 2024-06-15T07:34:15.759 INFO:teuthology.orchestra.run.smithi131.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.2-1443-g6f4b0956-1jammy [1,490 kB] 2024-06-15T07:34:15.789 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librados2. 2024-06-15T07:34:15.815 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../008-librados2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:15.855 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librados2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:15.989 INFO:teuthology.orchestra.run.smithi131.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.2-1443-g6f4b0956-1jammy [6,080 kB] 2024-06-15T07:34:16.275 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librbd1. 2024-06-15T07:34:16.302 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../009-librbd1_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:16.333 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librbd1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:16.780 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs2. 2024-06-15T07:34:16.807 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:16.846 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:16.869 INFO:teuthology.orchestra.run.smithi131.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.2-1443-g6f4b0956-1jammy [22.3 MB] 2024-06-15T07:34:17.115 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rados. 2024-06-15T07:34:17.132 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:17.173 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rados (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:17.510 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-15T07:34:17.537 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:17.576 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-argparse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:17.787 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cephfs. 2024-06-15T07:34:17.803 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:17.845 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cephfs (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:18.115 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-15T07:34:18.142 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:18.181 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:18.493 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-15T07:34:18.520 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-15T07:34:18.559 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T07:34:18.862 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-prettytable. 2024-06-15T07:34:18.889 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-06-15T07:34:18.929 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-15T07:34:19.181 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rbd. 2024-06-15T07:34:19.208 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:19.264 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rbd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:19.525 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-15T07:34:19.552 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-15T07:34:19.584 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T07:34:19.805 INFO:teuthology.orchestra.run.smithi131.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.2-1443-g6f4b0956-1jammy [14.1 kB] 2024-06-15T07:34:19.806 INFO:teuthology.orchestra.run.smithi131.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.2-1443-g6f4b0956-1jammy [831 kB] 2024-06-15T07:34:19.843 INFO:teuthology.orchestra.run.smithi131.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.2-1443-g6f4b0956-1jammy [2,154 kB] 2024-06-15T07:34:19.911 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librgw2. 2024-06-15T07:34:19.938 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../019-librgw2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:19.978 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librgw2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:20.091 INFO:teuthology.orchestra.run.smithi131.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.2-1443-g6f4b0956-1jammy [238 kB] 2024-06-15T07:34:20.125 INFO:teuthology.orchestra.run.smithi131.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.2-1443-g6f4b0956-1jammy [145 kB] 2024-06-15T07:34:20.128 INFO:teuthology.orchestra.run.smithi131.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.2-1443-g6f4b0956-1jammy [2,227 kB] 2024-06-15T07:34:20.380 INFO:teuthology.orchestra.run.smithi131.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.2-1443-g6f4b0956-1jammy [8,625 kB] 2024-06-15T07:34:20.481 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rgw. 2024-06-15T07:34:20.497 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:20.539 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:20.834 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-15T07:34:20.860 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-15T07:34:20.891 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T07:34:21.161 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libradosstriper1. 2024-06-15T07:34:21.188 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:21.227 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libradosstriper1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:21.306 INFO:teuthology.orchestra.run.smithi131.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.2-1443-g6f4b0956-1jammy [14.3 kB] 2024-06-15T07:34:21.307 INFO:teuthology.orchestra.run.smithi131.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.2-1443-g6f4b0956-1jammy [42.6 MB] 2024-06-15T07:34:21.697 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-common. 2024-06-15T07:34:21.724 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:21.764 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:23.653 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-base. 2024-06-15T07:34:23.680 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:23.721 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-base (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:24.266 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-15T07:34:24.292 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-15T07:34:24.332 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-15T07:34:24.467 INFO:teuthology.orchestra.run.smithi131.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.2-1443-g6f4b0956-1jammy [121 kB] 2024-06-15T07:34:24.469 INFO:teuthology.orchestra.run.smithi131.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.2-1443-g6f4b0956-1jammy [38.3 kB] 2024-06-15T07:34:24.470 INFO:teuthology.orchestra.run.smithi131.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.2-1443-g6f4b0956-1jammy [11.6 MB] 2024-06-15T07:34:24.576 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cheroot. 2024-06-15T07:34:24.603 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-15T07:34:24.635 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T07:34:24.938 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-15T07:34:24.953 INFO:teuthology.orchestra.run.smithi131.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-06-08-0902-reef/6f4b09560ed399c3f139402a02f389c2fc8bb2cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.2-1443-g6f4b0956-1jammy [93.1 kB] 2024-06-15T07:34:24.965 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-15T07:34:25.004 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-15T07:34:25.307 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-15T07:34:25.334 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-15T07:34:25.374 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-15T07:34:25.520 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 162 MB in 20s (8,277 kB/s) 2024-06-15T07:34:25.649 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libfmt8:amd64. 2024-06-15T07:34:25.660 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-15T07:34:25.687 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-15T07:34:25.726 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-15T07:34:25.747 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165982 files and directories currently installed.) 2024-06-15T07:34:25.752 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-06-15T07:34:25.831 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-06-15T07:34:26.005 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempora. 2024-06-15T07:34:26.032 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-06-15T07:34:26.071 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-15T07:34:26.159 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-15T07:34:26.186 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-15T07:34:26.226 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T07:34:26.374 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-portend. 2024-06-15T07:34:26.401 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-06-15T07:34:26.440 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-15T07:34:26.546 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-15T07:34:26.572 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-15T07:34:26.604 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T07:34:26.726 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-15T07:34:26.753 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-15T07:34:26.793 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-15T07:34:26.907 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-15T07:34:26.934 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-15T07:34:26.974 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T07:34:27.063 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-15T07:34:27.090 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-15T07:34:27.121 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-15T07:34:27.394 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-15T07:34:27.423 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-natsort. 2024-06-15T07:34:27.485 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-06-15T07:34:27.524 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-15T07:34:27.611 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:27.794 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-logutils. 2024-06-15T07:34:27.810 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-06-15T07:34:27.844 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-15T07:34:28.006 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-15T07:34:28.032 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:28.073 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:28.138 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-15T07:34:28.166 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-15T07:34:28.205 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-15T07:34:28.368 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-15T07:34:28.394 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T07:34:28.426 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:34:28.525 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-mako. 2024-06-15T07:34:28.552 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-15T07:34:28.591 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T07:34:28.754 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-15T07:34:28.781 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-15T07:34:28.821 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T07:34:28.903 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-15T07:34:28.930 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-15T07:34:28.969 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-15T07:34:29.141 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librados2. 2024-06-15T07:34:29.167 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../008-librados2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:29.216 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librados2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:29.263 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-15T07:34:29.279 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-15T07:34:29.314 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-15T07:34:29.591 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webob. 2024-06-15T07:34:29.618 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-15T07:34:29.658 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-15T07:34:29.660 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librbd1. 2024-06-15T07:34:29.676 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../009-librbd1_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:29.710 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librbd1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:30.011 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-waitress. 2024-06-15T07:34:30.038 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-15T07:34:30.080 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libcephfs2. 2024-06-15T07:34:30.081 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T07:34:30.106 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../010-libcephfs2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:30.147 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libcephfs2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempita. 2024-06-15T07:34:30.413 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-15T07:34:30.424 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rados. 2024-06-15T07:34:30.440 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../011-python3-rados_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:30.455 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T07:34:30.483 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rados (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:30.758 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-paste. 2024-06-15T07:34:30.785 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-15T07:34:30.802 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-15T07:34:30.825 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T07:34:30.828 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:30.868 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-ceph-argparse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:31.103 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-15T07:34:31.105 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cephfs. 2024-06-15T07:34:31.119 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-15T07:34:31.120 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../013-python3-cephfs_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:31.155 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cephfs (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:31.161 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T07:34:31.458 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-15T07:34:31.464 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-15T07:34:31.484 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:31.491 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-15T07:34:31.524 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:31.530 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-15T07:34:31.809 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webtest. 2024-06-15T07:34:31.835 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-06-15T07:34:31.836 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-15T07:34:31.862 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-15T07:34:31.875 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-15T07:34:31.902 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T07:34:32.128 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pecan. 2024-06-15T07:34:32.140 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-15T07:34:32.169 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T07:34:32.222 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-prettytable. 2024-06-15T07:34:32.249 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-06-15T07:34:32.289 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-15T07:34:32.449 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-certifi. 2024-06-15T07:34:32.476 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-06-15T07:34:32.507 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-15T07:34:32.508 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rbd. 2024-06-15T07:34:32.534 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../017-python3-rbd_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:32.567 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rbd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:32.835 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-idna. 2024-06-15T07:34:32.862 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-15T07:34:32.895 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-15T07:34:32.901 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T07:34:32.921 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-15T07:34:32.961 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T07:34:33.213 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-urllib3. 2024-06-15T07:34:33.240 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-15T07:34:33.279 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T07:34:33.406 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librgw2. 2024-06-15T07:34:33.433 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../019-librgw2_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:33.473 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librgw2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:33.799 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests. 2024-06-15T07:34:33.826 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-15T07:34:33.857 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T07:34:33.959 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rgw. 2024-06-15T07:34:33.985 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../020-python3-rgw_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:34.017 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:34.110 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-15T07:34:34.137 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-15T07:34:34.169 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T07:34:34.312 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-15T07:34:34.338 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-15T07:34:34.378 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T07:34:34.472 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-15T07:34:34.499 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:34.538 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:34.681 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libradosstriper1. 2024-06-15T07:34:34.708 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../022-libradosstriper1_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:34.748 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libradosstriper1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:34.893 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-15T07:34:34.909 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:34.943 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:35.127 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-common. 2024-06-15T07:34:35.153 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../023-ceph-common_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:35.193 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:35.246 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-15T07:34:35.273 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-15T07:34:35.312 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-15T07:34:35.582 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-openssl. 2024-06-15T07:34:35.609 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-06-15T07:34:35.648 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-15T07:34:35.993 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr. 2024-06-15T07:34:36.020 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../057-ceph-mgr_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:36.059 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:36.454 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mon. 2024-06-15T07:34:36.466 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-base. 2024-06-15T07:34:36.481 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../058-ceph-mon_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:36.493 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../024-ceph-base_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:36.521 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mon (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:36.542 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-base (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:37.049 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-osd. 2024-06-15T07:34:37.076 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../059-ceph-osd_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:37.107 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-osd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:37.112 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-15T07:34:37.139 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-15T07:34:37.178 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-15T07:34:38.294 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph. 2024-06-15T07:34:38.321 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../060-ceph_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:38.361 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:38.375 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cheroot. 2024-06-15T07:34:38.401 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-15T07:34:38.441 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T07:34:38.727 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-15T07:34:38.743 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-15T07:34:38.777 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-15T07:34:39.047 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-15T07:34:39.074 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-15T07:34:39.114 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-15T07:34:39.417 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-15T07:34:39.443 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-15T07:34:39.483 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-15T07:34:39.514 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-fuse. 2024-06-15T07:34:39.542 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../061-ceph-fuse_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:39.581 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:39.753 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-tempora. 2024-06-15T07:34:39.780 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-06-15T07:34:39.811 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-15T07:34:39.925 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mds. 2024-06-15T07:34:39.941 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../062-ceph-mds_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:39.975 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mds (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:40.089 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-portend. 2024-06-15T07:34:40.116 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-06-15T07:34:40.156 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-15T07:34:40.371 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package cephadm. 2024-06-15T07:34:40.398 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../063-cephadm_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:40.437 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:40.459 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-15T07:34:40.485 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-15T07:34:40.525 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-15T07:34:40.732 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-15T07:34:40.758 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-15T07:34:40.798 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-15T07:34:40.803 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-15T07:34:40.830 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-15T07:34:40.870 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-15T07:34:41.117 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-15T07:34:41.133 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T07:34:41.175 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T07:34:41.182 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-natsort. 2024-06-15T07:34:41.198 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-06-15T07:34:41.241 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-15T07:34:41.577 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-logutils. 2024-06-15T07:34:41.604 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-06-15T07:34:41.644 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-15T07:34:41.947 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-15T07:34:41.973 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-15T07:34:42.014 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-15T07:34:42.173 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-babel. 2024-06-15T07:34:42.189 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T07:34:42.222 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T07:34:42.275 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-mako. 2024-06-15T07:34:42.301 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-15T07:34:42.333 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T07:34:42.509 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jinja2. 2024-06-15T07:34:42.536 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-15T07:34:42.567 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T07:34:42.595 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-15T07:34:42.610 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-15T07:34:42.653 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-15T07:34:42.904 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-15T07:34:42.931 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:42.965 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-15T07:34:42.970 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:42.991 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-15T07:34:43.031 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-15T07:34:43.240 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pluggy. 2024-06-15T07:34:43.267 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-15T07:34:43.307 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-15T07:34:43.334 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-webob. 2024-06-15T07:34:43.360 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-15T07:34:43.400 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-15T07:34:43.618 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-py. 2024-06-15T07:34:43.645 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-06-15T07:34:43.684 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-15T07:34:43.720 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-waitress. 2024-06-15T07:34:43.747 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-15T07:34:43.789 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T07:34:44.022 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-15T07:34:44.049 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-15T07:34:44.065 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-tempita. 2024-06-15T07:34:44.092 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-15T07:34:44.123 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T07:34:44.271 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-15T07:34:44.419 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-paste. 2024-06-15T07:34:44.445 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-15T07:34:44.485 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T07:34:44.633 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-toml. 2024-06-15T07:34:44.660 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-06-15T07:34:44.699 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-15T07:34:44.780 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-15T07:34:44.795 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-15T07:34:44.838 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T07:34:44.960 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pytest. 2024-06-15T07:34:44.976 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-15T07:34:45.010 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T07:34:45.133 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-15T07:34:45.159 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-15T07:34:45.199 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-15T07:34:45.347 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-15T07:34:45.374 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-06-15T07:34:45.414 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-15T07:34:45.486 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-webtest. 2024-06-15T07:34:45.512 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-06-15T07:34:45.578 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-15T07:34:45.675 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-routes. 2024-06-15T07:34:45.694 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-15T07:34:45.725 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T07:34:45.889 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pecan. 2024-06-15T07:34:45.916 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-15T07:34:45.956 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T07:34:46.020 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-15T07:34:46.036 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:46.078 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:46.303 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-certifi. 2024-06-15T07:34:46.329 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-06-15T07:34:46.369 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-15T07:34:46.655 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-idna. 2024-06-15T07:34:46.682 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-15T07:34:46.730 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T07:34:47.042 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-urllib3. 2024-06-15T07:34:47.068 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-15T07:34:47.108 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T07:34:47.250 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-15T07:34:47.278 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-15T07:34:47.316 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T07:34:47.379 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-requests. 2024-06-15T07:34:47.405 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-15T07:34:47.436 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T07:34:47.734 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-15T07:34:47.760 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-15T07:34:47.800 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T07:34:47.829 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-joblib. 2024-06-15T07:34:47.856 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-15T07:34:47.895 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T07:34:48.086 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-15T07:34:48.102 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:48.245 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-modules-core (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:48.249 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-15T07:34:48.265 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-15T07:34:48.307 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-15T07:34:48.610 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn. 2024-06-15T07:34:48.617 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-15T07:34:48.637 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-15T07:34:48.644 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:48.676 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T07:34:48.683 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libsqlite3-mod-ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:48.953 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-15T07:34:48.979 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-15T07:34:49.020 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-15T07:34:49.322 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-openssl. 2024-06-15T07:34:49.349 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-06-15T07:34:49.389 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-15T07:34:49.401 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-15T07:34:49.429 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:49.468 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:49.734 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr. 2024-06-15T07:34:49.760 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../057-ceph-mgr_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:49.800 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:50.162 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mon. 2024-06-15T07:34:50.177 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../058-ceph-mon_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:50.212 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mon (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:50.454 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cachetools. 2024-06-15T07:34:50.483 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-06-15T07:34:50.521 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-15T07:34:50.791 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-osd. 2024-06-15T07:34:50.818 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../059-ceph-osd_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:50.858 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-osd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:50.883 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-15T07:34:50.911 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-15T07:34:50.941 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-15T07:34:51.220 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-15T07:34:51.248 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-15T07:34:51.278 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T07:34:51.523 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rsa. 2024-06-15T07:34:51.551 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-06-15T07:34:51.581 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-15T07:34:51.902 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-google-auth. 2024-06-15T07:34:51.920 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph. 2024-06-15T07:34:51.929 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-06-15T07:34:51.946 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../060-ceph_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:51.968 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-15T07:34:51.986 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:52.248 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-fuse. 2024-06-15T07:34:52.274 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../061-ceph-fuse_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:52.296 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-15T07:34:52.305 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:52.324 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-15T07:34:52.363 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T07:34:52.633 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-websocket. 2024-06-15T07:34:52.660 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-06-15T07:34:52.668 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mds. 2024-06-15T07:34:52.691 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-15T07:34:52.694 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../062-ceph-mds_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:52.734 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mds (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:52.986 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-15T07:34:53.013 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-15T07:34:53.052 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T07:34:53.087 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package cephadm. 2024-06-15T07:34:53.114 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../063-cephadm_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:53.145 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:53.432 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-15T07:34:53.458 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-15T07:34:53.498 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-15T07:34:53.749 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-15T07:34:53.766 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:53.808 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:53.810 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-15T07:34:53.826 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T07:34:53.868 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T07:34:54.127 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-15T07:34:54.155 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-15T07:34:54.193 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T07:34:54.597 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-15T07:34:54.625 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T07:34:54.672 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T07:34:54.899 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-babel. 2024-06-15T07:34:54.927 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T07:34:54.932 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package jq. 2024-06-15T07:34:54.957 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T07:34:54.961 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T07:34:54.999 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-15T07:34:55.228 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jinja2. 2024-06-15T07:34:55.254 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-15T07:34:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package socat. 2024-06-15T07:34:55.277 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-15T07:34:55.311 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T07:34:55.318 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-15T07:34:55.632 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package xmlstarlet. 2024-06-15T07:34:55.647 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-15T07:34:55.660 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-15T07:34:55.675 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:55.706 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-15T07:34:55.714 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:56.035 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pluggy. 2024-06-15T07:34:56.062 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-15T07:34:56.069 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-test. 2024-06-15T07:34:56.086 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:56.101 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-15T07:34:56.127 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-test (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:56.413 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-py. 2024-06-15T07:34:56.440 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-06-15T07:34:56.479 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-15T07:34:56.791 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-15T07:34:56.818 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-15T07:34:56.857 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-15T07:34:57.122 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-toml. 2024-06-15T07:34:57.135 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-06-15T07:34:57.169 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-15T07:34:57.414 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pytest. 2024-06-15T07:34:57.441 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-15T07:34:57.480 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T07:34:57.784 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-15T07:34:57.812 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-06-15T07:34:57.867 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-15T07:34:58.065 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-volume. 2024-06-15T07:34:58.093 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:58.112 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-routes. 2024-06-15T07:34:58.140 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-15T07:34:58.152 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-volume (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:58.179 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T07:34:58.452 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-15T07:34:58.469 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:34:58.474 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-15T07:34:58.490 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:34:58.510 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs-dev (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:58.524 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-dashboard (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:34:58.880 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package nvme-cli. 2024-06-15T07:34:58.908 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-15T07:34:59.047 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T07:34:59.578 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-15T07:34:59.606 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-15T07:34:59.718 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-15T07:34:59.879 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-15T07:34:59.895 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-15T07:34:59.938 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T07:35:00.146 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastescript. 2024-06-15T07:35:00.174 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-06-15T07:35:00.212 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-15T07:35:00.434 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-joblib. 2024-06-15T07:35:00.460 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-15T07:35:00.500 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T07:35:00.533 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-psutil. 2024-06-15T07:35:00.561 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-15T07:35:00.599 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-15T07:35:00.820 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-15T07:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-15T07:35:00.870 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-15T07:35:00.944 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pygments. 2024-06-15T07:35:00.972 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-15T07:35:01.018 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T07:35:01.140 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-sklearn. 2024-06-15T07:35:01.167 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-15T07:35:01.206 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T07:35:01.507 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-15T07:35:01.535 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-15T07:35:01.573 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-15T07:35:01.859 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplejson. 2024-06-15T07:35:01.887 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-15T07:35:01.925 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-15T07:35:01.951 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-15T07:35:01.979 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:35:02.024 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:02.186 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-15T07:35:02.203 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-15T07:35:02.236 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-15T07:35:02.882 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package radosgw. 2024-06-15T07:35:02.910 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../107-radosgw_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:02.948 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking radosgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:02.986 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cachetools. 2024-06-15T07:35:03.012 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-06-15T07:35:03.052 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-15T07:35:03.398 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-15T07:35:03.414 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-15T07:35:03.456 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-15T07:35:03.751 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package rbd-fuse. 2024-06-15T07:35:03.752 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-15T07:35:03.779 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:03.780 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-15T07:35:03.817 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking rbd-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:03.836 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T07:35:04.122 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-15T07:35:04.147 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rsa. 2024-06-15T07:35:04.174 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-06-15T07:35:04.213 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-15T07:35:04.472 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T07:35:04.526 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-google-auth. 2024-06-15T07:35:04.553 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-06-15T07:35:04.592 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-15T07:35:04.598 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T07:35:04.864 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-15T07:35:04.913 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-15T07:35:04.940 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-15T07:35:04.979 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T07:35:05.203 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-15T07:35:05.291 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-websocket. 2024-06-15T07:35:05.318 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-06-15T07:35:05.357 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-15T07:35:05.598 INFO:teuthology.orchestra.run.smithi012.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-15T07:35:05.669 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-15T07:35:05.675 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:05.696 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-15T07:35:05.736 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T07:35:06.134 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user cephadm....done 2024-06-15T07:35:06.256 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T07:35:06.358 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-15T07:35:06.374 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:35:06.416 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-k8sevents (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:06.524 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-15T07:35:06.728 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-15T07:35:06.755 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-15T07:35:06.775 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-15T07:35:06.786 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T07:35:06.901 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-15T07:35:07.123 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-15T07:35:07.150 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T07:35:07.152 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-15T07:35:07.189 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T07:35:07.400 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package jq. 2024-06-15T07:35:07.411 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T07:35:07.416 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T07:35:07.450 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-15T07:35:07.538 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-15T07:35:07.703 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package socat. 2024-06-15T07:35:07.730 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-15T07:35:07.770 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-15T07:35:07.848 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T07:35:08.082 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package xmlstarlet. 2024-06-15T07:35:08.097 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-15T07:35:08.140 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-15T07:35:08.208 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-15T07:35:08.468 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-15T07:35:08.495 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-test. 2024-06-15T07:35:08.522 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../096-ceph-test_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:08.561 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-test (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:08.727 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-15T07:35:08.978 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-15T07:35:09.254 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-15T07:35:09.514 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-argparse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:09.764 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T07:35:09.891 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-15T07:35:10.142 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T07:35:10.268 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-15T07:35:10.557 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-volume. 2024-06-15T07:35:10.582 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../097-ceph-volume_18.2.2-1443-g6f4b0956-1jammy_all.deb ... 2024-06-15T07:35:10.623 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-volume (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:10.635 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T07:35:10.761 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-15T07:35:10.903 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-15T07:35:10.930 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:10.969 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libcephfs-dev (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:11.038 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T07:35:11.314 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-15T07:35:11.339 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package nvme-cli. 2024-06-15T07:35:11.367 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-15T07:35:11.406 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T07:35:11.615 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T07:35:11.753 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-15T07:35:11.778 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-15T07:35:11.819 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-15T07:35:12.231 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pastescript. 2024-06-15T07:35:12.259 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-06-15T07:35:12.291 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T07:35:12.297 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-15T07:35:12.526 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T07:35:12.585 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-psutil. 2024-06-15T07:35:12.613 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-15T07:35:12.643 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-15T07:35:12.652 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-15T07:35:12.778 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T07:35:12.955 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pygments. 2024-06-15T07:35:12.982 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-15T07:35:13.021 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T07:35:13.079 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-15T07:35:13.480 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-15T07:35:13.500 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-15T07:35:13.528 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-15T07:35:13.567 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-15T07:35:13.740 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T07:35:13.828 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-simplejson. 2024-06-15T07:35:13.844 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-15T07:35:13.878 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-15T07:35:14.133 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T07:35:14.139 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-15T07:35:14.167 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-15T07:35:14.205 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-15T07:35:14.426 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T07:35:14.769 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-15T07:35:14.968 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package radosgw. 2024-06-15T07:35:14.996 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../107-radosgw_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:15.020 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-15T07:35:15.034 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking radosgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:15.146 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:15.439 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T07:35:15.748 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T07:35:15.771 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package rbd-fuse. 2024-06-15T07:35:15.787 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../108-rbd-fuse_18.2.2-1443-g6f4b0956-1jammy_amd64.deb ... 2024-06-15T07:35:15.829 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking rbd-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:16.092 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-15T07:35:16.359 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T07:35:16.485 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T07:35:16.747 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-15T07:35:16.987 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:17.064 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-15T07:35:17.121 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-15T07:35:17.381 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T07:35:17.473 INFO:teuthology.orchestra.run.smithi131.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-15T07:35:17.507 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-15T07:35:17.546 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:17.633 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T07:35:17.943 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-15T07:35:18.062 INFO:teuthology.orchestra.run.smithi131.stdout:Adding system user cephadm....done 2024-06-15T07:35:18.194 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-15T07:35:18.211 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T07:35:18.445 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:18.520 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-15T07:35:18.554 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-06-15T07:35:18.655 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-15T07:35:18.763 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-15T07:35:18.873 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-15T07:35:18.948 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-15T07:35:19.124 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-15T07:35:19.207 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-15T07:35:19.375 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T07:35:19.467 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-15T07:35:19.502 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-15T07:35:19.718 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-15T07:35:19.811 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T07:35:19.952 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-15T07:35:20.147 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-15T07:35:20.407 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-15T07:35:20.470 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T07:35:20.596 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-15T07:35:20.683 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-15T07:35:20.880 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T07:35:20.934 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-15T07:35:20.990 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T07:35:21.194 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-15T07:35:21.299 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-15T07:35:21.445 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-ceph-argparse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:21.609 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-15T07:35:21.679 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T07:35:21.780 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-15T07:35:21.868 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T07:35:22.015 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T07:35:22.141 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-15T07:35:22.237 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-15T07:35:22.488 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T07:35:22.501 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T07:35:22.589 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T07:35:22.627 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-15T07:35:22.903 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T07:35:22.965 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-15T07:35:23.162 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-15T07:35:23.225 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-15T07:35:23.455 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T07:35:23.484 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:23.585 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T07:35:23.903 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T07:35:24.149 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T07:35:24.204 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T07:35:24.392 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T07:35:24.472 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-15T07:35:24.598 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-15T07:35:24.710 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-15T07:35:24.874 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-15T07:35:24.894 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T07:35:25.150 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T07:35:25.204 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-15T07:35:25.409 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-15T07:35:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-15T07:35:25.777 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-15T07:35:26.031 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T07:35:26.112 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T07:35:26.383 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T07:35:26.456 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T07:35:26.582 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librados2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:26.651 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T07:35:26.708 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librgw2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:26.834 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:26.960 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:26.994 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-15T07:35:27.086 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libradosstriper1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:27.245 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-15T07:35:27.289 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-15T07:35:27.371 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:27.565 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librbd1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:27.664 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T07:35:27.691 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-modules-core (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:27.801 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:27.966 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T07:35:28.002 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T07:35:28.002 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T07:35:28.485 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs-dev (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:28.612 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rados (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:28.738 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T07:35:29.064 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:29.173 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-15T07:35:29.416 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T07:35:29.525 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-15T07:35:29.645 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T07:35:29.930 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-15T07:35:29.992 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rbd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:30.118 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up rbd-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:30.172 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-15T07:35:30.244 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:30.408 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cephfs (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:30.415 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:30.513 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:30.532 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-06-15T07:35:30.692 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-15T07:35:30.993 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-15T07:35:30.996 INFO:teuthology.orchestra.run.smithi012.stdout:Adding group ceph....done 2024-06-15T07:35:31.261 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-15T07:35:31.383 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user ceph....done 2024-06-15T07:35:31.512 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-15T07:35:31.550 INFO:teuthology.orchestra.run.smithi012.stdout:Setting system user ceph properties....done 2024-06-15T07:35:31.569 INFO:teuthology.orchestra.run.smithi012.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-15T07:35:31.712 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-15T07:35:31.763 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-15T07:35:32.005 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-15T07:35:32.039 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-15T07:35:32.257 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T07:35:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-15T07:35:32.506 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-test (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:32.631 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up radosgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:32.659 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T07:35:32.784 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T07:35:33.094 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-15T07:35:33.099 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T07:35:33.099 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T07:35:33.412 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-15T07:35:33.566 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-base (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:33.671 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T07:35:33.899 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-15T07:35:34.052 INFO:teuthology.orchestra.run.smithi131.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-15T07:35:34.324 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T07:35:34.377 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mds (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:34.450 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T07:35:34.602 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T07:35:34.602 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T07:35:34.835 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-15T07:35:35.079 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:35.103 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-15T07:35:35.297 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T07:35:35.297 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T07:35:35.362 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T07:35:35.488 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T07:35:35.797 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T07:35:35.889 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-osd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:36.073 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T07:35:36.204 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T07:35:36.204 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T07:35:36.449 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-15T07:35:36.575 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-15T07:35:36.699 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:36.825 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:36.843 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-15T07:35:36.976 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mon (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:37.153 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T07:35:37.197 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T07:35:37.197 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T07:35:37.420 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-15T07:35:37.711 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:37.805 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-15T07:35:37.853 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:37.979 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-dashboard (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:38.073 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T07:35:38.131 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-volume (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:38.391 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T07:35:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-15T07:35:38.517 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librados2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:38.643 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librgw2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:38.778 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libsqlite3-mod-ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:38.929 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libcephfs2 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:39.055 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libradosstriper1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:39.257 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-15T07:35:39.534 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librbd1 (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:39.660 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-modules-core (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:39.684 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-15T07:35:39.786 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:39.999 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T07:35:39.999 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T07:35:40.166 INFO:teuthology.orchestra.run.smithi012.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-15T07:35:40.172 DEBUG:teuthology.parallel:result is None 2024-06-15T07:35:40.454 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libcephfs-dev (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:40.580 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rados (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:40.698 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T07:35:41.918 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rbd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:42.053 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up rbd-fuse (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:42.179 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:42.305 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cephfs (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:42.431 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-common (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:42.924 INFO:teuthology.orchestra.run.smithi131.stdout:Adding group ceph....done 2024-06-15T07:35:43.292 INFO:teuthology.orchestra.run.smithi131.stdout:Adding system user ceph....done 2024-06-15T07:35:43.451 INFO:teuthology.orchestra.run.smithi131.stdout:Setting system user ceph properties....done 2024-06-15T07:35:43.466 INFO:teuthology.orchestra.run.smithi131.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-15T07:35:43.602 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-15T07:35:43.939 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-15T07:35:44.666 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-test (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up radosgw (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:45.268 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T07:35:45.269 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T07:35:45.743 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-base (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:46.056 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-15T07:35:46.656 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mds (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:46.877 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T07:35:46.878 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T07:35:47.332 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:47.551 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T07:35:47.551 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T07:35:48.025 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-osd (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:48.303 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T07:35:48.303 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T07:35:48.769 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-k8sevents (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:48.878 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-diskprediction-local (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:49.012 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mon (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:49.224 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T07:35:49.224 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T07:35:49.680 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-cephadm (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:49.823 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:49.949 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-dashboard (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:50.092 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-volume (18.2.2-1443-g6f4b0956-1jammy) ... 2024-06-15T07:35:50.428 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-15T07:35:51.562 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-15T07:35:52.524 INFO:teuthology.orchestra.run.smithi131.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-15T07:35:52.529 DEBUG:teuthology.parallel:result is None 2024-06-15T07:35:52.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T07:35:52.748 DEBUG:teuthology.orchestra.run.smithi012:> dpkg-query -W -f '${Version}' ceph 2024-06-15T07:35:52.768 INFO:teuthology.orchestra.run.smithi012.stdout:18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:35:52.769 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:35:52.769 INFO:teuthology.task.install:The correct ceph version 18.2.2-1443-g6f4b0956-1jammy is installed. 2024-06-15T07:35:52.771 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T07:35:52.978 DEBUG:teuthology.orchestra.run.smithi131:> dpkg-query -W -f '${Version}' ceph 2024-06-15T07:35:52.997 INFO:teuthology.orchestra.run.smithi131.stdout:18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:35:52.997 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1443-g6f4b0956-1jammy 2024-06-15T07:35:52.997 INFO:teuthology.task.install:The correct ceph version 18.2.2-1443-g6f4b0956-1jammy is installed. 2024-06-15T07:35:52.999 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-15T07:35:52.999 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:52.999 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-15T07:35:53.013 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:53.014 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-15T07:35:53.051 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-15T07:35:53.051 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:53.051 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-06-15T07:35:53.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-15T07:35:53.125 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:53.125 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2024-06-15T07:35:53.137 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-15T07:35:53.190 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-15T07:35:53.190 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:53.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-15T07:35:53.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-15T07:35:53.259 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:53.260 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-15T07:35:53.272 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-15T07:35:53.327 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-15T07:35:53.327 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:53.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-06-15T07:35:53.340 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-15T07:35:53.396 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:53.396 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2024-06-15T07:35:53.409 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-15T07:35:53.463 INFO:teuthology.run_tasks:Running task ceph... 2024-06-15T07:35:53.530 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-15T07:35:53.531 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2024-06-15T07:35:53.533 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /var/log/ceph 2024-06-15T07:35:53.545 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-15T07:35:53.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-15T07:35:53.589 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-15T07:35:53.601 INFO:tasks.ceph:Creating extra log directories... 2024-06-15T07:35:53.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-15T07:35:53.649 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-15T07:35:53.671 INFO:tasks.ceph:Creating ceph cluster c1... 2024-06-15T07:35:53.671 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-15T07:35:53.671 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785', 'branch': 'wip-yuri5-testing-2024-06-08-0902-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7756785', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-06-08-0902-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-reef-drop-centos8', 'sha1': 'e3daf20513a97cf7004cd9e7bfdba8f5a015acbc'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8659, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-reef-drop-centos8', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': 'e3daf20513a97cf7004cd9e7bfdba8f5a015acbc', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBY/mxvVX7P4IG2Dm/bOwz9MRbKtI9BfQX0UPeVZGLZFeDIHNpYIKG/jJFJzA+qwA+Zwf8C+QeH61+6wTjTpg8=', 'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9YO8YSxAcQpNBx9QAf6bw6L9BwqGOqgW0B3R9qnUbclUzl56KIW9mLr5HJMLZcyc2Xbm5Uw2X2CHr7R2sCq0Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c1b06c2b5d3a02a3583265a115f09e052855bcf2', 'timestamp': '2024-06-14_18:51:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-15T07:35:53.671 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-06-15T07:35:53.705 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-06-15T07:35:53.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-06-15T07:35:53.754 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/run/ceph 2024-06-15T07:35:53.768 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:53.768 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T07:35:53.812 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T07:35:53.813 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-06-15 07:30:37.017873128 +0000 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-06-15 07:30:36.773875307 +0000 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-06-15 07:30:36.773875307 +0000 2024-06-15T07:35:53.861 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-06-15T07:35:53.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T07:35:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-06-15T07:35:53.915 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-06-15T07:35:53.916 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000368078 s, 1.4 MB/s 2024-06-15T07:35:53.917 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T07:35:53.966 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-06-15T07:35:54.012 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-06-15 07:30:37.517868662 +0000 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-06-15 07:30:37.257870984 +0000 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-06-15 07:30:37.257870984 +0000 2024-06-15T07:35:54.013 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-06-15T07:35:54.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T07:35:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-06-15T07:35:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-06-15T07:35:54.067 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00043654 s, 1.2 MB/s 2024-06-15T07:35:54.068 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T07:35:54.118 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-06-15T07:35:54.164 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T07:35:54.164 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-06-15 07:30:38.005864302 +0000 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-06-15 07:30:37.749866590 +0000 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-06-15 07:30:37.749866590 +0000 2024-06-15T07:35:54.165 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-06-15T07:35:54.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T07:35:54.219 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-06-15T07:35:54.219 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-06-15T07:35:54.219 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000356415 s, 1.4 MB/s 2024-06-15T07:35:54.220 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T07:35:54.270 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-06-15T07:35:54.317 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T07:35:54.317 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:35:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-06-15T07:35:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:35:54.317 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-06-15 07:30:38.521859694 +0000 2024-06-15T07:35:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-06-15 07:30:38.249862123 +0000 2024-06-15T07:35:54.318 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-06-15 07:30:38.249862123 +0000 2024-06-15T07:35:54.318 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2024-06-15T07:35:54.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T07:35:54.371 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-06-15T07:35:54.371 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-06-15T07:35:54.371 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.0003872 s, 1.3 MB/s 2024-06-15T07:35:54.373 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T07:35:54.422 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:35:54.422 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-06-15T07:35:54.422 INFO:tasks.ceph:Generating config... 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] debug rgw = 20 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-06-15T07:35:54.424 INFO:tasks.ceph:[client] setgroup = ceph 2024-06-15T07:35:54.425 INFO:tasks.ceph:[client] setuser = ceph 2024-06-15T07:35:54.425 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-06-15T07:35:54.425 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-15T07:35:54.425 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-15T07:35:54.425 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-15T07:35:54.425 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-15T07:35:54.425 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-06-15T07:35:54.425 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-06-15T07:35:54.426 INFO:tasks.ceph:Setting up c1.mon.a... 2024-06-15T07:35:54.426 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-06-15T07:35:54.498 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.keyring 2024-06-15T07:35:54.514 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-06-15T07:35:54.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-06-15T07:35:54.647 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.12')] 2024-06-15T07:35:54.648 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-06-15T07:35:54.648 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:54.648 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-15T07:35:54.692 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --print /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:35:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:35:54.771 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid d85db544-8771-4673-b5a0-b9593832e8be 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = pacific 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:fsid d85db544-8771-4673-b5a0-b9593832e8be 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2024-06-15T07:35:54.774039+0000 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-06-15T07:35:54.774039+0000 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 16 (pacific) 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2024-06-15T07:35:54.772 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-06-15T07:35:54.773 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-06-15T07:35:54.774 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-15T07:35:54.820 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID d85db544-8771-4673-b5a0-b9593832e8be... 2024-06-15T07:35:54.820 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-06-15T07:35:54.865 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.894 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-15T07:35:54.895 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-15T07:35:54.896 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.12 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = d85db544-8771-4673-b5a0-b9593832e8be 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.897 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-06-15T07:35:54.898 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: rocksdb delete range threshold = 0 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:35:54.899 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-06-15T07:35:54.900 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:35:54.901 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw = 20 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw sync = 20 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt require ssl = False 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms backend = testing 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw curl low speed time = 300 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data log num shards = 4 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw md log max shards = 4 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync data inject err probability = 0.1 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync log trim interval = 0 2024-06-15T07:35:54.902 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync meta inject err probability = 0.1 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync obj etag verify = True 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: setgroup = ceph 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: setuser = ceph 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data notify interval msec = 0 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 1000 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-06-15T07:35:54.903 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10000 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: chdir = "" 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: auth supported = cephx 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: filestore xattr use omap = true 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: mon clock drift allowed = 1.000 2024-06-15T07:35:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: osd crush chooseleaf type = 0 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: auth debug = true 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on old message = true 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on bug = true 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: mon pg warn max object skew = 0 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: # disable pg_autoscaler by default for new pools 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default size = 2 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow primary affinity = true 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow pg remap = true 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on legacy crush tunables = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on crush straw calc version zero = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on no sortbitwise = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on osd down out interval zero = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on too few osds = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: mon_allow_pool_size_one = true 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: osd default data pool replay window = 5 2024-06-15T07:35:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: mon allow pool delete = true 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: mon cluster log file level = debug 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: debug asserts on shutdown = true 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: mon health detail to clog = false 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: mon host = 172.21.15.12 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = d85db544-8771-4673-b5a0-b9593832e8be 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout:[osd] 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: osd journal size = 100 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub load threshold = 5.0 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub max interval = 600 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: osd mclock profile = high_recovery_ops 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.907 INFO:teuthology.orchestra.run.smithi131.stdout: osd recover clone overlap = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd recovery max chunk = 1048576 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug shutdown = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug op order = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug verify stray on activate = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd open classes on start = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug pg log writeout = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd deep scrub update digest min age = 30 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: osd map max advance = 10 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: journal zero on create = true 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.908 INFO:teuthology.orchestra.run.smithi131.stdout: filestore ondisk finisher threads = 3 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: filestore apply finisher threads = 3 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: bdev debug aio = true 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug misdirected ops = true 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug osd = 20 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: rocksdb delete range threshold = 0 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr] 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug mgr = 20 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:35:54.909 INFO:teuthology.orchestra.run.smithi131.stdout: mgr/telemetry/nag = false 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout:[mon] 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: debug paxos = 20 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon data avail warn = 5 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon mgr mkfs grace = 240 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd reporter subtree level = osd 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd prime pg temp = true 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-15T07:35:54.910 INFO:teuthology.orchestra.run.smithi131.stdout: auth mon ticket ttl = 660 # 11m 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: auth service ticket ttl = 240 # 4m 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: # don't complain about insecure global_id in the test suite 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: # 1m isn't quite enough 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: mon_down_mkfs_grace = 2m 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_filestore_osds = false 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout:[client] 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: rgw cache enabled = true 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable ops log = true 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable usage log = true 2024-06-15T07:35:54.911 INFO:teuthology.orchestra.run.smithi131.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw = 20 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw sync = 20 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt require ssl = False 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms backend = testing 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw curl low speed time = 300 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data log num shards = 4 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw md log max shards = 4 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync data inject err probability = 0.1 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync log trim interval = 0 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync meta inject err probability = 0.1 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync obj etag verify = True 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: setgroup = ceph 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout: setuser = ceph 2024-06-15T07:35:54.912 INFO:teuthology.orchestra.run.smithi131.stdout:[mon.a] 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data notify interval msec = 0 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 1000 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2024-06-15T07:35:54.913 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10000 2024-06-15T07:35:54.914 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-06-15T07:35:54.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-06-15T07:35:55.002 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-15T07:35:55.002 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:55.002 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-06-15T07:35:55.008 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:55.008 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-06-15T07:35:55.056 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:35:55.056 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:55.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/c1.keyring 2024-06-15T07:35:55.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-06-15T07:35:55.121 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:35:55.121 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-06-15T07:35:55.169 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:35:55.169 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:55.169 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/c1.keyring 2024-06-15T07:35:55.169 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-06-15T07:35:55.192 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:35:55.192 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-06-15T07:35:55.240 INFO:tasks.ceph:Setting up mon nodes... 2024-06-15T07:35:55.240 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-15T07:35:55.240 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-06-15T07:35:55.285 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-06-15T07:35:55.288 INFO:tasks.ceph:Setting up mds nodes... 2024-06-15T07:35:55.288 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-15T07:35:55.288 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-06-15T07:35:55.320 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.client.0.keyring 2024-06-15T07:35:55.330 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-06-15T07:35:55.403 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.client.1.keyring 2024-06-15T07:35:55.413 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-15T07:35:55.413 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-06-15T07:35:55.414 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-06-15T07:35:55.466 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:35:55.466 INFO:tasks.ceph:role: c1.osd.0 2024-06-15T07:35:55.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:35:55.466 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:35:55.528 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:35:55.529 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:35:55.529 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:35:56.000 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-06-15T07:35:56.003 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-06-15T07:35:56.003 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-06-15T07:35:56.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-06-15T07:35:56.105 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:35:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:35:56.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-06-15T07:35:56.160 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:35:56.160 INFO:tasks.ceph:role: c1.osd.1 2024-06-15T07:35:56.160 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:35:56.160 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:35:56.225 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:35:56.690 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-06-15T07:35:56.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-06-15T07:35:56.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-06-15T07:35:56.846 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-06-15T07:35:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:35:56.858 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:35:56.858 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-06-15T07:35:56.912 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:35:56.912 INFO:tasks.ceph:role: c1.osd.2 2024-06-15T07:35:56.912 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:35:56.912 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-15T07:35:56.976 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:35:56.976 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:35:56.977 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:35:57.445 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2024-06-15T07:35:57.448 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2024-06-15T07:35:57.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-06-15T07:35:57.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-06-15T07:35:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:35:57.588 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:35:57.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:35:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-06-15T07:35:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-06-15T07:35:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-06-15T07:35:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-06-15T07:35:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-06-15T07:35:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:35:57.719+0000 7f9c8f41c7c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-06-15T07:35:59.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-06-15T07:35:59.937 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:36:00.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.015+0000 7f03782c57c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-06-15T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.019+0000 7f03782c57c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-06-15T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.019+0000 7f03782c57c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-06-15T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.019+0000 7f03782c57c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-06-15T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.019+0000 7f03782c57c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-06-15T07:36:00.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:00.019+0000 7f03782c57c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-06-15T07:36:02.209 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-06-15T07:36:02.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:36:02.351 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.347+0000 7fde9d6137c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-06-15T07:36:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.351+0000 7fde9d6137c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-06-15T07:36:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.351+0000 7fde9d6137c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-06-15T07:36:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.351+0000 7fde9d6137c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-06-15T07:36:02.352 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.351+0000 7fde9d6137c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-06-15T07:36:02.353 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:36:02.351+0000 7fde9d6137c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-06-15T07:36:04.539 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-06-15T07:36:04.593 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-15T07:36:04.593 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-06-15T07:36:04.648 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.648 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-06-15T07:36:04.704 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.705 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-06-15T07:36:04.761 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.761 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-06-15T07:36:04.816 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.816 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-06-15T07:36:04.864 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:04.864 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-06-15T07:36:04.912 INFO:tasks.ceph:Adding keys to all mons... 2024-06-15T07:36:04.912 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/c1.keyring 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBbRG1msXsgERAA8hVsntKR4pMmqZtwcYyXyw== 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBdRG1m6lwoKxAAN/xTt5wX3OieYAfabFUJrw== 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBgRG1m8+NHARAAKr6VswqptYbWDcrA2KO1IA== 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBiRG1mVI4QFRAAY5ijwvy27gz6rj53k8QLvg== 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBbRG1m7GY6ExAA/t93s4PyL/AyFQq9CBYMfg== 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2024-06-15T07:36:04.967 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBbRG1mWpssGBAAbOyAcTuA+sRW6xC2Jvh60Q== 2024-06-15T07:36:04.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-15T07:36:05.058 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:05.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:05.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:05.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-15T07:36:05.361 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-15T07:36:05.403 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-15T07:36:05.403 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-06-15T07:36:05.461 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-06-15T07:36:05.557 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-06-15T07:36:05.569 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-06-15T07:36:05.616 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-06-15T07:36:05.616 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-06-15T07:36:05.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-06-15T07:36:05.661 INFO:tasks.ceph.c1.mon.a:Started 2024-06-15T07:36:05.661 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-06-15T07:36:05.662 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-06-15T07:36:05.662 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-06-15T07:36:05.667 INFO:tasks.ceph.c1.mgr.x:Started 2024-06-15T07:36:05.667 INFO:tasks.ceph:Setting crush tunables to default 2024-06-15T07:36:05.667 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd crush tunables default 2024-06-15T07:36:05.935 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-06-15T07:36:05.953 INFO:tasks.ceph:check_enable_crimson: False 2024-06-15T07:36:05.953 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-06-15T07:36:05.953 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:05.953 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-06-15T07:36:05.963 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:05.963 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-06-15T07:36:06.018 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:06.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-06-15T07:36:06.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new 4388352a-075b-473b-b02a-c059eea43d5d 0 2024-06-15T07:36:06.076 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:06.071+0000 7f416025c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-15T07:36:06.283 INFO:teuthology.orchestra.run.smithi012.stdout:0 2024-06-15T07:36:06.301 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new 531ba675-284c-4299-982c-350b64c9d388 1 2024-06-15T07:36:06.501 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-06-15T07:36:06.519 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new 85f1f8b5-7b09-4301-9113-12db3f8e88c1 2 2024-06-15T07:36:06.598 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:06.595+0000 7f416025c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-15T07:36:06.724 INFO:teuthology.orchestra.run.smithi012.stdout:2 2024-06-15T07:36:06.741 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-06-15T07:36:06.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-06-15T07:36:06.743 INFO:tasks.ceph.c1.osd.0:Started 2024-06-15T07:36:06.743 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-06-15T07:36:06.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-06-15T07:36:06.745 INFO:tasks.ceph.c1.osd.1:Started 2024-06-15T07:36:06.746 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-06-15T07:36:06.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-06-15T07:36:06.747 INFO:tasks.ceph.c1.osd.2:Started 2024-06-15T07:36:06.748 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-06-15T07:36:06.802 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:06.799+0000 7f416025c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-15T07:36:06.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:06.946 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"fsid":"d85db544-8771-4673-b5a0-b9593832e8be","created":"2024-06-15T07:36:05.805625+0000","modified":"2024-06-15T07:36:06.724706+0000","last_up_change":"0.000000","last_in_change":"2024-06-15T07:36:06.724706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4388352a-075b-473b-b02a-c059eea43d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"531ba675-284c-4299-982c-350b64c9d388","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"85f1f8b5-7b09-4301-9113-12db3f8e88c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:06.962 INFO:tasks.ceph.ceph_manager.c1:[] 2024-06-15T07:36:06.963 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-15T07:36:07.264 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-06-15T07:36:07.373 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2024-06-15T07:36:07.371+0000 7f2e0287c7c0 -1 Falling back to public interface 2024-06-15T07:36:07.377 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2024-06-15T07:36:07.375+0000 7febb2d957c0 -1 Falling back to public interface 2024-06-15T07:36:07.381 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2024-06-15T07:36:07.379+0000 7fe5fe0147c0 -1 Falling back to public interface 2024-06-15T07:36:07.387 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.383+0000 7f416025c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-15T07:36:07.445 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.443+0000 7f416025c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-15T07:36:07.448 INFO:teuthology.misc.health.smithi012.stdout: 2024-06-15T07:36:07.448 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":5,"fsid":"d85db544-8771-4673-b5a0-b9593832e8be","created":"2024-06-15T07:36:05.805625+0000","modified":"2024-06-15T07:36:06.724706+0000","last_up_change":"0.000000","last_in_change":"2024-06-15T07:36:06.724706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4388352a-075b-473b-b02a-c059eea43d5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"531ba675-284c-4299-982c-350b64c9d388","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"85f1f8b5-7b09-4301-9113-12db3f8e88c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:07.463 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-06-15T07:36:07.501 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.499+0000 7f416025c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-15T07:36:07.582 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.579+0000 7f416025c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-15T07:36:07.636 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.635+0000 7f416025c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-15T07:36:07.717 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.715+0000 7f416025c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-15T07:36:07.771 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.767+0000 7f416025c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-15T07:36:07.928 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.923+0000 7f416025c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-15T07:36:07.990 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:07.987+0000 7f416025c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-15T07:36:08.657 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-06-15T07:36:08.657 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-06-15T07:36:08.657 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr: from numpy import show_config as show_numpy_config 2024-06-15T07:36:08.719 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:08.715+0000 7f416025c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-15T07:36:08.907 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:08.903+0000 7f416025c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-15T07:36:09.082 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:09.079+0000 7f416025c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-15T07:36:09.152 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:09.151+0000 7f416025c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-15T07:36:09.303 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:09.299+0000 7f416025c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-15T07:36:09.958 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:09.955+0000 7f416025c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-15T07:36:10.036 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:10.035+0000 7f416025c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-15T07:36:10.226 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:10.223+0000 7f416025c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-15T07:36:10.303 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:10.299+0000 7f416025c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-15T07:36:10.432 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2024-06-15T07:36:10.431+0000 7fe5fe0147c0 -1 osd.1 0 log_to_monitors true 2024-06-15T07:36:10.433 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2024-06-15T07:36:10.431+0000 7febb2d957c0 -1 osd.0 0 log_to_monitors true 2024-06-15T07:36:10.455 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2024-06-15T07:36:10.451+0000 7f2e0287c7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T07:36:10.611 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:10.607+0000 7f416025c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-15T07:36:10.676 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:10.671+0000 7f416025c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-15T07:36:11.743 INFO:tasks.ceph.c1.mgr.x.smithi012.stderr:2024-06-15T07:36:11.739+0000 7f4107812640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-15T07:36:12.882 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2024-06-15T07:36:12.879+0000 7fe5f9fb9640 -1 osd.1 0 waiting for initial osdmap 2024-06-15T07:36:12.921 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2024-06-15T07:36:12.919+0000 7fe5f1dec640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:12.921 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2024-06-15T07:36:12.919+0000 7f2dfe821640 -1 osd.2 0 waiting for initial osdmap 2024-06-15T07:36:12.939 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2024-06-15T07:36:12.935+0000 7febaed3a640 -1 osd.0 0 waiting for initial osdmap 2024-06-15T07:36:12.953 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2024-06-15T07:36:12.951+0000 7f2df6654640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:12.973 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2024-06-15T07:36:12.971+0000 7feba6b6d640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:13.766 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-06-15T07:36:14.063 INFO:teuthology.misc.health.smithi012.stdout: 2024-06-15T07:36:14.063 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":8,"fsid":"d85db544-8771-4673-b5a0-b9593832e8be","created":"2024-06-15T07:36:05.805625+0000","modified":"2024-06-15T07:36:13.924052+0000","last_up_change":"2024-06-15T07:36:13.924052+0000","last_in_change":"2024-06-15T07:36:06.724706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4388352a-075b-473b-b02a-c059eea43d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":23001},{"type":"v1","addr":"172.21.15.12:6809","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":23001},{"type":"v1","addr":"172.21.15.12:6811","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":23001},{"type":"v1","addr":"172.21.15.12:6815","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":23001},{"type":"v1","addr":"172.21.15.12:6813","nonce":23001}]},"public_addr":"172.21.15.12:6809/23001","cluster_addr":"172.21.15.12:6811/23001","heartbeat_back_addr":"172.21.15.12:6815/23001","heartbeat_front_addr":"172.21.15.12:6813/23001","state":["exists","up"]},{"osd":1,"uuid":"531ba675-284c-4299-982c-350b64c9d388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":23000},{"type":"v1","addr":"172.21.15.12:6817","nonce":23000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":23000},{"type":"v1","addr":"172.21.15.12:6819","nonce":23000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":23000},{"type":"v1","addr":"172.21.15.12:6823","nonce":23000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":23000},{"type":"v1","addr":"172.21.15.12:6821","nonce":23000}]},"public_addr":"172.21.15.12:6817/23000","cluster_addr":"172.21.15.12:6819/23000","heartbeat_back_addr":"172.21.15.12:6823/23000","heartbeat_front_addr":"172.21.15.12:6821/23000","state":["exists","up"]},{"osd":2,"uuid":"85f1f8b5-7b09-4301-9113-12db3f8e88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":23018},{"type":"v1","addr":"172.21.15.12:6801","nonce":23018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":23018},{"type":"v1","addr":"172.21.15.12:6803","nonce":23018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":23018},{"type":"v1","addr":"172.21.15.12:6807","nonce":23018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":23018},{"type":"v1","addr":"172.21.15.12:6805","nonce":23018}]},"public_addr":"172.21.15.12:6801/23018","cluster_addr":"172.21.15.12:6803/23018","heartbeat_back_addr":"172.21.15.12:6807/23018","heartbeat_front_addr":"172.21.15.12:6805/23018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:14.081 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-06-15T07:36:14.081 INFO:tasks.ceph:Creating RBD pool 2024-06-15T07:36:14.081 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd pool create rbd 8 2024-06-15T07:36:14.933 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2024-06-15T07:36:14.959 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-15T07:36:15.935 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-15T07:36:15.970 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-06-15T07:36:15.970 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-06-15T07:36:15.970 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-06-15T07:36:15.970 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-06-15T07:36:15.971 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-15T07:36:16.328 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:16.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":22824},{"type":"v1","addr":"172.21.15.12:6825","nonce":22824}]},"active_addr":"172.21.15.12:6825/22824","active_change":"2024-06-15T07:36:10.699742+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":877688546}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1752810704}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":4159653030}]}]} 2024-06-15T07:36:16.346 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-06-15T07:36:16.346 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-06-15T07:36:16.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-06-15T07:36:16.663 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:16.663 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"fsid":"d85db544-8771-4673-b5a0-b9593832e8be","created":"2024-06-15T07:36:05.805625+0000","modified":"2024-06-15T07:36:15.929076+0000","last_up_change":"2024-06-15T07:36:13.924052+0000","last_in_change":"2024-06-15T07:36:06.724706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-06-15T07:36:14.418830+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-06-15T07:36:14.779207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4388352a-075b-473b-b02a-c059eea43d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":23001},{"type":"v1","addr":"172.21.15.12:6809","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":23001},{"type":"v1","addr":"172.21.15.12:6811","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":23001},{"type":"v1","addr":"172.21.15.12:6815","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":23001},{"type":"v1","addr":"172.21.15.12:6813","nonce":23001}]},"public_addr":"172.21.15.12:6809/23001","cluster_addr":"172.21.15.12:6811/23001","heartbeat_back_addr":"172.21.15.12:6815/23001","heartbeat_front_addr":"172.21.15.12:6813/23001","state":["exists","up"]},{"osd":1,"uuid":"531ba675-284c-4299-982c-350b64c9d388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":23000},{"type":"v1","addr":"172.21.15.12:6817","nonce":23000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":23000},{"type":"v1","addr":"172.21.15.12:6819","nonce":23000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":23000},{"type":"v1","addr":"172.21.15.12:6823","nonce":23000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":23000},{"type":"v1","addr":"172.21.15.12:6821","nonce":23000}]},"public_addr":"172.21.15.12:6817/23000","cluster_addr":"172.21.15.12:6819/23000","heartbeat_back_addr":"172.21.15.12:6823/23000","heartbeat_front_addr":"172.21.15.12:6821/23000","state":["exists","up"]},{"osd":2,"uuid":"85f1f8b5-7b09-4301-9113-12db3f8e88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":23018},{"type":"v1","addr":"172.21.15.12:6801","nonce":23018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":23018},{"type":"v1","addr":"172.21.15.12:6803","nonce":23018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":23018},{"type":"v1","addr":"172.21.15.12:6807","nonce":23018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":23018},{"type":"v1","addr":"172.21.15.12:6805","nonce":23018}]},"public_addr":"172.21.15.12:6801/23018","cluster_addr":"172.21.15.12:6803/23018","heartbeat_back_addr":"172.21.15.12:6807/23018","heartbeat_front_addr":"172.21.15.12:6805/23018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.387772+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.435097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.482894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:16.682 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-06-15T07:36:16.682 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-06-15T07:36:16.974 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:16.975 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"fsid":"d85db544-8771-4673-b5a0-b9593832e8be","created":"2024-06-15T07:36:05.805625+0000","modified":"2024-06-15T07:36:16.931045+0000","last_up_change":"2024-06-15T07:36:13.924052+0000","last_in_change":"2024-06-15T07:36:06.724706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-06-15T07:36:14.418830+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-06-15T07:36:14.779207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4388352a-075b-473b-b02a-c059eea43d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":23001},{"type":"v1","addr":"172.21.15.12:6809","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":23001},{"type":"v1","addr":"172.21.15.12:6811","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":23001},{"type":"v1","addr":"172.21.15.12:6815","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":23001},{"type":"v1","addr":"172.21.15.12:6813","nonce":23001}]},"public_addr":"172.21.15.12:6809/23001","cluster_addr":"172.21.15.12:6811/23001","heartbeat_back_addr":"172.21.15.12:6815/23001","heartbeat_front_addr":"172.21.15.12:6813/23001","state":["exists","up"]},{"osd":1,"uuid":"531ba675-284c-4299-982c-350b64c9d388","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":23000},{"type":"v1","addr":"172.21.15.12:6817","nonce":23000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":23000},{"type":"v1","addr":"172.21.15.12:6819","nonce":23000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":23000},{"type":"v1","addr":"172.21.15.12:6823","nonce":23000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":23000},{"type":"v1","addr":"172.21.15.12:6821","nonce":23000}]},"public_addr":"172.21.15.12:6817/23000","cluster_addr":"172.21.15.12:6819/23000","heartbeat_back_addr":"172.21.15.12:6823/23000","heartbeat_front_addr":"172.21.15.12:6821/23000","state":["exists","up"]},{"osd":2,"uuid":"85f1f8b5-7b09-4301-9113-12db3f8e88c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":23018},{"type":"v1","addr":"172.21.15.12:6801","nonce":23018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":23018},{"type":"v1","addr":"172.21.15.12:6803","nonce":23018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":23018},{"type":"v1","addr":"172.21.15.12:6807","nonce":23018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":23018},{"type":"v1","addr":"172.21.15.12:6805","nonce":23018}]},"public_addr":"172.21.15.12:6801/23018","cluster_addr":"172.21.15.12:6803/23018","heartbeat_back_addr":"172.21.15.12:6807/23018","heartbeat_front_addr":"172.21.15.12:6805/23018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.387772+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.435097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:11.482894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:16.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-06-15T07:36:16.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-06-15T07:36:16.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-06-15T07:36:17.147 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2024-06-15T07:36:17.148 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-06-15T07:36:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:17.150 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-06-15T07:36:17.150 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:17.151 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-06-15T07:36:17.430 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:17.448 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2024-06-15T07:36:17.475 INFO:teuthology.orchestra.run.smithi012.stdout:34359738370 2024-06-15T07:36:17.478 INFO:teuthology.orchestra.run.smithi012.stdout:34359738370 2024-06-15T07:36:17.495 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2024-06-15T07:36:17.497 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.1 2024-06-15T07:36:18.449 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-06-15T07:36:18.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-06-15T07:36:18.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-06-15T07:36:18.755 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:18.773 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2024-06-15T07:36:18.786 INFO:teuthology.orchestra.run.smithi012.stdout:34359738370 2024-06-15T07:36:18.788 INFO:teuthology.orchestra.run.smithi012.stdout:34359738370 2024-06-15T07:36:18.805 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.2 2024-06-15T07:36:18.806 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.1 2024-06-15T07:36:19.774 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-06-15T07:36:19.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-06-15T07:36:19.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-06-15T07:36:20.080 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2024-06-15T07:36:20.097 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2024-06-15T07:36:20.097 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:20.111 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:20.117 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2024-06-15T07:36:20.133 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2024-06-15T07:36:20.133 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:20.140 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2024-06-15T07:36:20.140 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:20.140 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-06-15T07:36:20.140 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-06-15T07:36:20.456 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:20.456 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-06-15T07:36:20.473 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-06-15T07:36:18.709433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83232,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489568,"statfs":{"total":322122547200,"available":322037317632,"internally_reserved":0,"allocated":1454080,"data_stored":1247365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941067+0000","last_change":"2024-06-15T07:36:15.948158+0000","last_active":"2024-06-15T07:36:16.941067+0000","last_peered":"2024-06-15T07:36:16.941067+0000","last_clean":"2024-06-15T07:36:16.941067+0000","last_became_active":"2024-06-15T07:36:15.947878+0000","last_became_peered":"2024-06-15T07:36:15.947878+0000","last_unstale":"2024-06-15T07:36:16.941067+0000","last_undegraded":"2024-06-15T07:36:16.941067+0000","last_fullsized":"2024-06-15T07:36:16.941067+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:10:46.279736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941099+0000","last_change":"2024-06-15T07:36:15.948592+0000","last_active":"2024-06-15T07:36:16.941099+0000","last_peered":"2024-06-15T07:36:16.941099+0000","last_clean":"2024-06-15T07:36:16.941099+0000","last_became_active":"2024-06-15T07:36:15.948224+0000","last_became_peered":"2024-06-15T07:36:15.948224+0000","last_unstale":"2024-06-15T07:36:16.941099+0000","last_undegraded":"2024-06-15T07:36:16.941099+0000","last_fullsized":"2024-06-15T07:36:16.941099+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:09:16.230248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:15.948816+0000","last_change":"2024-06-15T07:36:15.946737+0000","last_active":"2024-06-15T07:36:15.948816+0000","last_peered":"2024-06-15T07:36:15.948816+0000","last_clean":"2024-06-15T07:36:15.948816+0000","last_became_active":"2024-06-15T07:36:15.946369+0000","last_became_peered":"2024-06-15T07:36:15.946369+0000","last_unstale":"2024-06-15T07:36:15.948816+0000","last_undegraded":"2024-06-15T07:36:15.948816+0000","last_fullsized":"2024-06-15T07:36:15.948816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:41:32.964826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.940913+0000","last_change":"2024-06-15T07:36:15.947720+0000","last_active":"2024-06-15T07:36:16.940913+0000","last_peered":"2024-06-15T07:36:16.940913+0000","last_clean":"2024-06-15T07:36:16.940913+0000","last_became_active":"2024-06-15T07:36:15.947363+0000","last_became_peered":"2024-06-15T07:36:15.947363+0000","last_unstale":"2024-06-15T07:36:16.940913+0000","last_undegraded":"2024-06-15T07:36:16.940913+0000","last_fullsized":"2024-06-15T07:36:16.940913+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:23:17.696945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.504037+0000","last_change":"2024-06-15T07:36:15.946684+0000","last_active":"2024-06-15T07:36:16.504037+0000","last_peered":"2024-06-15T07:36:16.504037+0000","last_clean":"2024-06-15T07:36:16.504037+0000","last_became_active":"2024-06-15T07:36:15.946387+0000","last_became_peered":"2024-06-15T07:36:15.946387+0000","last_unstale":"2024-06-15T07:36:16.504037+0000","last_undegraded":"2024-06-15T07:36:16.504037+0000","last_fullsized":"2024-06-15T07:36:16.504037+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T18:43:36.437491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941008+0000","last_change":"2024-06-15T07:36:15.948104+0000","last_active":"2024-06-15T07:36:16.941008+0000","last_peered":"2024-06-15T07:36:16.941008+0000","last_clean":"2024-06-15T07:36:16.941008+0000","last_became_active":"2024-06-15T07:36:15.947742+0000","last_became_peered":"2024-06-15T07:36:15.947742+0000","last_unstale":"2024-06-15T07:36:16.941008+0000","last_undegraded":"2024-06-15T07:36:16.941008+0000","last_fullsized":"2024-06-15T07:36:16.941008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:00:48.052665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:15.948625+0000","last_change":"2024-06-15T07:36:15.944591+0000","last_active":"2024-06-15T07:36:15.948625+0000","last_peered":"2024-06-15T07:36:15.948625+0000","last_clean":"2024-06-15T07:36:15.948625+0000","last_became_active":"2024-06-15T07:36:15.944094+0000","last_became_peered":"2024-06-15T07:36:15.944094+0000","last_unstale":"2024-06-15T07:36:15.948625+0000","last_undegraded":"2024-06-15T07:36:15.948625+0000","last_fullsized":"2024-06-15T07:36:15.948625+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T18:02:11.837529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:15.948791+0000","last_change":"2024-06-15T07:36:15.947115+0000","last_active":"2024-06-15T07:36:15.948791+0000","last_peered":"2024-06-15T07:36:15.948791+0000","last_clean":"2024-06-15T07:36:15.948791+0000","last_became_active":"2024-06-15T07:36:15.946753+0000","last_became_peered":"2024-06-15T07:36:15.946753+0000","last_unstale":"2024-06-15T07:36:15.948791+0000","last_undegraded":"2024-06-15T07:36:15.948791+0000","last_fullsized":"2024-06-15T07:36:15.948791+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:23:17.696945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.940937+0000","last_change":"2024-06-15T07:36:15.945551+0000","last_active":"2024-06-15T07:36:16.940937+0000","last_peered":"2024-06-15T07:36:16.940937+0000","last_clean":"2024-06-15T07:36:16.940937+0000","last_became_active":"2024-06-15T07:36:15.945138+0000","last_became_peered":"2024-06-15T07:36:15.945138+0000","last_unstale":"2024-06-15T07:36:16.940937+0000","last_undegraded":"2024-06-15T07:36:16.940937+0000","last_fullsized":"2024-06-15T07:36:16.940937+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:41:32.964826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28188,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829412,"statfs":{"total":107374182400,"available":107345317888,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27940,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829660,"statfs":{"total":107374182400,"available":107345571840,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27104,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830496,"statfs":{"total":107374182400,"available":107346427904,"internally_reserved":0,"allocated":86016,"data_stored":21469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T07:36:20.474 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-06-15T07:36:20.746 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:20.746 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-06-15T07:36:20.762 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-06-15T07:36:20.709857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83504,"kb_used_data":1436,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489296,"statfs":{"total":322122547200,"available":322037039104,"internally_reserved":0,"allocated":1470464,"data_stored":1255061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941067+0000","last_change":"2024-06-15T07:36:15.948158+0000","last_active":"2024-06-15T07:36:16.941067+0000","last_peered":"2024-06-15T07:36:16.941067+0000","last_clean":"2024-06-15T07:36:16.941067+0000","last_became_active":"2024-06-15T07:36:15.947878+0000","last_became_peered":"2024-06-15T07:36:15.947878+0000","last_unstale":"2024-06-15T07:36:16.941067+0000","last_undegraded":"2024-06-15T07:36:16.941067+0000","last_fullsized":"2024-06-15T07:36:16.941067+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:10:46.279736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941099+0000","last_change":"2024-06-15T07:36:15.948592+0000","last_active":"2024-06-15T07:36:16.941099+0000","last_peered":"2024-06-15T07:36:16.941099+0000","last_clean":"2024-06-15T07:36:16.941099+0000","last_became_active":"2024-06-15T07:36:15.948224+0000","last_became_peered":"2024-06-15T07:36:15.948224+0000","last_unstale":"2024-06-15T07:36:16.941099+0000","last_undegraded":"2024-06-15T07:36:16.941099+0000","last_fullsized":"2024-06-15T07:36:16.941099+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:09:16.230248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:15.948816+0000","last_change":"2024-06-15T07:36:15.946737+0000","last_active":"2024-06-15T07:36:15.948816+0000","last_peered":"2024-06-15T07:36:15.948816+0000","last_clean":"2024-06-15T07:36:15.948816+0000","last_became_active":"2024-06-15T07:36:15.946369+0000","last_became_peered":"2024-06-15T07:36:15.946369+0000","last_unstale":"2024-06-15T07:36:15.948816+0000","last_undegraded":"2024-06-15T07:36:15.948816+0000","last_fullsized":"2024-06-15T07:36:15.948816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:41:32.964826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.940913+0000","last_change":"2024-06-15T07:36:15.947720+0000","last_active":"2024-06-15T07:36:16.940913+0000","last_peered":"2024-06-15T07:36:16.940913+0000","last_clean":"2024-06-15T07:36:16.940913+0000","last_became_active":"2024-06-15T07:36:15.947363+0000","last_became_peered":"2024-06-15T07:36:15.947363+0000","last_unstale":"2024-06-15T07:36:16.940913+0000","last_undegraded":"2024-06-15T07:36:16.940913+0000","last_fullsized":"2024-06-15T07:36:16.940913+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:23:17.696945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.504037+0000","last_change":"2024-06-15T07:36:15.946684+0000","last_active":"2024-06-15T07:36:16.504037+0000","last_peered":"2024-06-15T07:36:16.504037+0000","last_clean":"2024-06-15T07:36:16.504037+0000","last_became_active":"2024-06-15T07:36:15.946387+0000","last_became_peered":"2024-06-15T07:36:15.946387+0000","last_unstale":"2024-06-15T07:36:16.504037+0000","last_undegraded":"2024-06-15T07:36:16.504037+0000","last_fullsized":"2024-06-15T07:36:16.504037+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T18:43:36.437491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.941008+0000","last_change":"2024-06-15T07:36:15.948104+0000","last_active":"2024-06-15T07:36:16.941008+0000","last_peered":"2024-06-15T07:36:16.941008+0000","last_clean":"2024-06-15T07:36:16.941008+0000","last_became_active":"2024-06-15T07:36:15.947742+0000","last_became_peered":"2024-06-15T07:36:15.947742+0000","last_unstale":"2024-06-15T07:36:16.941008+0000","last_undegraded":"2024-06-15T07:36:16.941008+0000","last_fullsized":"2024-06-15T07:36:16.941008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:00:48.052665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":10,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:17.129616+0000","last_change":"2024-06-15T07:36:15.944591+0000","last_active":"2024-06-15T07:36:17.129616+0000","last_peered":"2024-06-15T07:36:17.129616+0000","last_clean":"2024-06-15T07:36:17.129616+0000","last_became_active":"2024-06-15T07:36:15.944094+0000","last_became_peered":"2024-06-15T07:36:15.944094+0000","last_unstale":"2024-06-15T07:36:17.129616+0000","last_undegraded":"2024-06-15T07:36:17.129616+0000","last_fullsized":"2024-06-15T07:36:17.129616+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T18:02:11.837529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-06-15T07:36:15.948791+0000","last_change":"2024-06-15T07:36:15.947115+0000","last_active":"2024-06-15T07:36:15.948791+0000","last_peered":"2024-06-15T07:36:15.948791+0000","last_clean":"2024-06-15T07:36:15.948791+0000","last_became_active":"2024-06-15T07:36:15.946753+0000","last_became_peered":"2024-06-15T07:36:15.946753+0000","last_unstale":"2024-06-15T07:36:15.948791+0000","last_undegraded":"2024-06-15T07:36:15.948791+0000","last_fullsized":"2024-06-15T07:36:15.948791+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:23:17.696945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:16.940937+0000","last_change":"2024-06-15T07:36:15.945551+0000","last_active":"2024-06-15T07:36:16.940937+0000","last_peered":"2024-06-15T07:36:16.940937+0000","last_clean":"2024-06-15T07:36:16.940937+0000","last_became_active":"2024-06-15T07:36:15.945138+0000","last_became_peered":"2024-06-15T07:36:15.945138+0000","last_unstale":"2024-06-15T07:36:16.940937+0000","last_undegraded":"2024-06-15T07:36:16.940937+0000","last_fullsized":"2024-06-15T07:36:16.940937+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:14.925028+0000","last_clean_scrub_stamp":"2024-06-15T07:36:14.925028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:41:32.964826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28188,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829412,"statfs":{"total":107374182400,"available":107345317888,"internally_reserved":0,"allocated":684032,"data_stored":612948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28200,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829400,"statfs":{"total":107374182400,"available":107345305600,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":92,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":94208,"data_stored":25317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T07:36:20.763 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-06-15T07:36:20.763 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-06-15T07:36:20.763 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-06-15T07:36:20.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-06-15T07:36:21.068 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:21.068 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-15T07:36:21.085 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-06-15T07:36:21.086 INFO:teuthology.run_tasks:Running task ceph... 2024-06-15T07:36:21.095 INFO:tasks.ceph:Creating ceph cluster c2... 2024-06-15T07:36:21.095 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-15T07:36:21.095 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785', 'branch': 'wip-yuri5-testing-2024-06-08-0902-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7756785', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-06-08-0902-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-reef-drop-centos8', 'sha1': 'e3daf20513a97cf7004cd9e7bfdba8f5a015acbc'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 8659, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-reef-drop-centos8', 'suite_path': '/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/cbodley/ceph.git', 'suite_sha1': 'e3daf20513a97cf7004cd9e7bfdba8f5a015acbc', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGBY/mxvVX7P4IG2Dm/bOwz9MRbKtI9BfQX0UPeVZGLZFeDIHNpYIKG/jJFJzA+qwA+Zwf8C+QeH61+6wTjTpg8=', 'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9YO8YSxAcQpNBx9QAf6bw6L9BwqGOqgW0B3R9qnUbclUzl56KIW9mLr5HJMLZcyc2Xbm5Uw2X2CHr7R2sCq0Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c1b06c2b5d3a02a3583265a115f09e052855bcf2', 'timestamp': '2024-06-14_18:51:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-15T07:36:21.096 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-06-15T07:36:21.098 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-06-15T07:36:21.104 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2024-06-15T07:36:21.145 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/run/ceph 2024-06-15T07:36:21.160 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:21.160 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T07:36:21.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T07:36:21.208 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-06-15T07:36:21.255 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-06-15 07:30:37.056241524 +0000 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-06-15 07:30:36.772244059 +0000 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-06-15 07:30:36.772244059 +0000 2024-06-15T07:36:21.256 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-06-15T07:36:21.256 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T07:36:21.309 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-06-15T07:36:21.310 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-06-15T07:36:21.310 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000394841 s, 1.3 MB/s 2024-06-15T07:36:21.311 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T07:36:21.360 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-06-15 07:30:37.564236993 +0000 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-06-15 07:30:37.304239312 +0000 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-06-15 07:30:37.304239312 +0000 2024-06-15T07:36:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-06-15T07:36:21.410 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T07:36:21.462 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-06-15T07:36:21.462 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-06-15T07:36:21.462 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000430983 s, 1.2 MB/s 2024-06-15T07:36:21.463 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T07:36:21.513 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-06-15T07:36:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T07:36:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:36:21.559 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-06-15T07:36:21.559 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:36:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-06-15 07:30:38.040232747 +0000 2024-06-15T07:36:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-06-15 07:30:37.816234744 +0000 2024-06-15T07:36:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-06-15 07:30:37.816234744 +0000 2024-06-15T07:36:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-06-15T07:36:21.560 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T07:36:21.614 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-06-15T07:36:21.614 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-06-15T07:36:21.614 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.00044815 s, 1.1 MB/s 2024-06-15T07:36:21.615 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T07:36:21.664 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-06-15T07:36:21.711 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-06-15 07:30:38.508228572 +0000 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-06-15 07:30:38.272230676 +0000 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-06-15 07:30:38.272230676 +0000 2024-06-15T07:36:21.712 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-06-15T07:36:21.712 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T07:36:21.765 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-06-15T07:36:21.765 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-06-15T07:36:21.766 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.0003411 s, 1.5 MB/s 2024-06-15T07:36:21.767 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T07:36:21.816 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:36:21.816 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-06-15T07:36:21.817 INFO:tasks.ceph:Generating config... 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] debug rgw = 20 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-06-15T07:36:21.818 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-06-15T07:36:21.819 INFO:tasks.ceph:[client] setgroup = ceph 2024-06-15T07:36:21.819 INFO:tasks.ceph:[client] setuser = ceph 2024-06-15T07:36:21.819 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-06-15T07:36:21.819 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-15T07:36:21.819 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-15T07:36:21.819 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-15T07:36:21.819 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-15T07:36:21.819 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-06-15T07:36:21.819 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-06-15T07:36:21.819 INFO:tasks.ceph:Setting up c2.mon.a... 2024-06-15T07:36:21.820 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-06-15T07:36:21.890 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.keyring 2024-06-15T07:36:21.908 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-06-15T07:36:21.986 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-06-15T07:36:22.039 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.131')] 2024-06-15T07:36:22.039 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.131'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-06-15T07:36:22.040 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:22.040 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-15T07:36:22.088 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.131 --print /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:22.165 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:36:22.165 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: generated fsid cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:setting min_mon_release = pacific 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:epoch 0 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:fsid cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:last_changed 2024-06-15T07:36:22.168965+0000 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:created 2024-06-15T07:36:22.168965+0000 2024-06-15T07:36:22.166 INFO:teuthology.orchestra.run.smithi131.stdout:min_mon_release 16 (pacific) 2024-06-15T07:36:22.167 INFO:teuthology.orchestra.run.smithi131.stdout:election_strategy: 1 2024-06-15T07:36:22.167 INFO:teuthology.orchestra.run.smithi131.stdout:0: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.a 2024-06-15T07:36:22.167 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-06-15T07:36:22.168 DEBUG:teuthology.orchestra.run.smithi131:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-15T07:36:22.216 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee... 2024-06-15T07:36:22.216 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-06-15T07:36:22.219 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-06-15T07:36:22.261 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-06-15T07:36:22.261 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2024-06-15T07:36:22.262 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2024-06-15T07:36:22.263 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2024-06-15T07:36:22.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.131 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.265 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.266 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: rocksdb delete range threshold = 0 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2024-06-15T07:36:22.267 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2024-06-15T07:36:22.268 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.269 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw = 20 2024-06-15T07:36:22.270 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw sync = 20 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt require ssl = False 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms backend = testing 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw curl low speed time = 300 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data log num shards = 4 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw md log max shards = 4 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync data inject err probability = 0.1 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync log trim interval = 0 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync meta inject err probability = 0.1 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync obj etag verify = True 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: setgroup = ceph 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout: setuser = ceph 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2024-06-15T07:36:22.271 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data notify interval msec = 0 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 1000 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2024-06-15T07:36:22.272 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10000 2024-06-15T07:36:22.289 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: chdir = "" 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: auth supported = cephx 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: filestore xattr use omap = true 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: mon clock drift allowed = 1.000 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: osd crush chooseleaf type = 0 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: auth debug = true 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on old message = true 2024-06-15T07:36:22.290 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on bug = true 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon pg warn max object skew = 0 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: # disable pg_autoscaler by default for new pools 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default size = 2 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow primary affinity = true 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow pg remap = true 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on legacy crush tunables = false 2024-06-15T07:36:22.291 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on crush straw calc version zero = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on no sortbitwise = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on osd down out interval zero = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on too few osds = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon_allow_pool_size_one = true 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: osd default data pool replay window = 5 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: mon allow pool delete = true 2024-06-15T07:36:22.292 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: mon cluster log file level = debug 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: debug asserts on shutdown = true 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: mon health detail to clog = false 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: mon host = 172.21.15.131 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.293 INFO:teuthology.orchestra.run.smithi131.stdout:[osd] 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd journal size = 100 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub load threshold = 5.0 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub max interval = 600 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd mclock profile = high_recovery_ops 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd recover clone overlap = true 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd recovery max chunk = 1048576 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug shutdown = true 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug op order = true 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug verify stray on activate = true 2024-06-15T07:36:22.294 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: osd open classes on start = true 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug pg log writeout = true 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: osd deep scrub update digest min age = 30 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: osd map max advance = 10 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: journal zero on create = true 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.295 INFO:teuthology.orchestra.run.smithi131.stdout: filestore ondisk finisher threads = 3 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: filestore apply finisher threads = 3 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: bdev debug aio = true 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug misdirected ops = true 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug osd = 20 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: rocksdb delete range threshold = 0 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr] 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug mgr = 20 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-06-15T07:36:22.296 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mgr/telemetry/nag = false 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout:[mon] 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: debug paxos = 20 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon data avail warn = 5 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon mgr mkfs grace = 240 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd reporter subtree level = osd 2024-06-15T07:36:22.297 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd prime pg temp = true 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: auth mon ticket ttl = 660 # 11m 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: auth service ticket ttl = 240 # 4m 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: # don't complain about insecure global_id in the test suite 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: # 1m isn't quite enough 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: mon_down_mkfs_grace = 2m 2024-06-15T07:36:22.298 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_filestore_osds = false 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout:[client] 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: rgw cache enabled = true 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable ops log = true 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable usage log = true 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw = 20 2024-06-15T07:36:22.299 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw sync = 20 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt require ssl = False 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms backend = testing 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw curl low speed time = 300 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data log num shards = 4 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw md log max shards = 4 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync data inject err probability = 0.1 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync log trim interval = 0 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync meta inject err probability = 0.1 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync obj etag verify = True 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: setgroup = ceph 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout: setuser = ceph 2024-06-15T07:36:22.300 INFO:teuthology.orchestra.run.smithi131.stdout:[mon.a] 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data notify interval msec = 0 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 1000 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2024-06-15T07:36:22.301 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10000 2024-06-15T07:36:22.302 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-06-15T07:36:22.302 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-06-15T07:36:22.393 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-15T07:36:22.393 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:22.393 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-06-15T07:36:22.399 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:22.399 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-06-15T07:36:22.451 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T07:36:22.452 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:22.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/c2.keyring 2024-06-15T07:36:22.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-06-15T07:36:22.476 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-06-15T07:36:22.477 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:22.525 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:36:22.525 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:22.525 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/c2.keyring 2024-06-15T07:36:22.525 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-06-15T07:36:22.548 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:22.548 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:22.596 INFO:tasks.ceph:Setting up mon nodes... 2024-06-15T07:36:22.596 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-15T07:36:22.596 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-06-15T07:36:22.681 INFO:teuthology.orchestra.run.smithi131.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-06-15T07:36:22.683 INFO:tasks.ceph:Setting up mds nodes... 2024-06-15T07:36:22.683 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-15T07:36:22.684 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-06-15T07:36:22.759 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.client.0.keyring 2024-06-15T07:36:22.771 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-06-15T07:36:22.850 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.client.1.keyring 2024-06-15T07:36:22.860 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-15T07:36:22.860 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-06-15T07:36:22.860 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-06-15T07:36:22.914 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:36:22.914 INFO:tasks.ceph:role: c2.osd.0 2024-06-15T07:36:22.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:36:22.914 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:36:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:36:22.976 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:36:22.976 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:36:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:36:22.976 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:36:23.422 INFO:teuthology.orchestra.run.smithi131.stdout:Discarding blocks...Done. 2024-06-15T07:36:23.424 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2024-06-15T07:36:23.424 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-06-15T07:36:23.508 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-06-15T07:36:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:36:23.566 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:36:23.566 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-06-15T07:36:23.619 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:36:23.619 INFO:tasks.ceph:role: c2.osd.1 2024-06-15T07:36:23.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:36:23.619 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-15T07:36:23.682 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:36:23.683 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:36:24.178 INFO:teuthology.orchestra.run.smithi131.stdout:Discarding blocks...Done. 2024-06-15T07:36:24.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2024-06-15T07:36:24.180 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-06-15T07:36:24.326 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-06-15T07:36:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:36:24.337 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:36:24.337 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-06-15T07:36:24.395 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-06-15T07:36:24.395 INFO:tasks.ceph:role: c2.osd.2 2024-06-15T07:36:24.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T07:36:24.395 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-15T07:36:24.459 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2024-06-15T07:36:24.460 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-15T07:36:24.460 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-06-15T07:36:24.460 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-15T07:36:24.460 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-15T07:36:24.939 INFO:teuthology.orchestra.run.smithi131.stdout:Discarding blocks...Done. 2024-06-15T07:36:24.941 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2024-06-15T07:36:24.942 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-06-15T07:36:25.079 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-06-15T07:36:25.090 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2024-06-15T07:36:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T07:36:25.091 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:25.206 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.205+0000 7f05305947c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-06-15T07:36:25.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.209+0000 7f05305947c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-06-15T07:36:25.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.209+0000 7f05305947c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-06-15T07:36:25.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.209+0000 7f05305947c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-06-15T07:36:25.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.209+0000 7f05305947c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-06-15T07:36:25.208 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:25.209+0000 7f05305947c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-06-15T07:36:27.376 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-06-15T07:36:27.390 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:27.469 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-06-15T07:36:27.470 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-06-15T07:36:27.470 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-06-15T07:36:27.470 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-06-15T07:36:27.470 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-06-15T07:36:27.471 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:27.469+0000 7fed7dd717c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-06-15T07:36:29.638 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-06-15T07:36:29.692 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:29.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.773+0000 7f162c46a7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-06-15T07:36:29.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.773+0000 7f162c46a7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-06-15T07:36:29.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.773+0000 7f162c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-06-15T07:36:29.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.773+0000 7f162c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-06-15T07:36:29.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.773+0000 7f162c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-06-15T07:36:29.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:36:29.777+0000 7f162c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-06-15T07:36:31.958 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-06-15T07:36:32.012 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-15T07:36:32.012 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.012 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-06-15T07:36:32.066 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.066 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-06-15T07:36:32.118 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.118 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-06-15T07:36:32.170 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.170 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-06-15T07:36:32.222 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.223 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-06-15T07:36:32.271 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:32.271 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-06-15T07:36:32.319 INFO:tasks.ceph:Adding keys to all mons... 2024-06-15T07:36:32.320 DEBUG:teuthology.orchestra.run.smithi131:> sudo tee -a /etc/ceph/c2.keyring 2024-06-15T07:36:32.373 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr.x] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB2RG1mxY/IKBAAWqfhFBwa2mtY5HGhmd0GZQ== 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB5RG1meFR7DBAAkMvyns7ZACyja1HCQDQhjw== 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB7RG1mIZsvHBAAVP2lTRAvIDB2z7vNdckXgw== 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB9RG1m+bRULhAAmsEgQwlnk3opBJEv5qnKTg== 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB2RG1mt6d2LRAA0dU8hWwX8sYxYFBcqN8T6w== 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout:[client.1] 2024-06-15T07:36:32.374 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQB2RG1mm+LZMhAAEOqs28QHjbp35jqHeZCCZQ== 2024-06-15T07:36:32.375 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-15T07:36:32.464 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:32.548 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:32.637 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-15T07:36:32.677 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-15T07:36:32.760 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-15T07:36:32.845 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-15T07:36:32.845 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-06-15T07:36:32.898 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-06-15T07:36:32.987 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-06-15T07:36:32.997 DEBUG:teuthology.orchestra.run.smithi131:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-06-15T07:36:33.042 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-06-15T07:36:33.042 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-06-15T07:36:33.042 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-06-15T07:36:33.084 INFO:tasks.ceph.c2.mon.a:Started 2024-06-15T07:36:33.084 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-06-15T07:36:33.084 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-06-15T07:36:33.084 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-06-15T07:36:33.086 INFO:tasks.ceph.c2.mgr.x:Started 2024-06-15T07:36:33.086 INFO:tasks.ceph:Setting crush tunables to default 2024-06-15T07:36:33.087 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd crush tunables default 2024-06-15T07:36:33.313 INFO:teuthology.orchestra.run.smithi131.stderr:adjusted tunables profile to default 2024-06-15T07:36:33.331 INFO:tasks.ceph:check_enable_crimson: False 2024-06-15T07:36:33.331 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-06-15T07:36:33.332 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:33.332 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-06-15T07:36:33.343 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:33.343 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-06-15T07:36:33.396 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-06-15T07:36:33.396 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-06-15T07:36:33.436 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:33.437+0000 7f7ef80bc180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-15T07:36:33.446 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new 27b386e8-8bb6-43c9-bb3f-a7f27057c5bb 0 2024-06-15T07:36:33.674 INFO:teuthology.orchestra.run.smithi131.stdout:0 2024-06-15T07:36:33.690 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new 7a28a428-45ad-45ca-8ff7-3234b551efe9 1 2024-06-15T07:36:33.881 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-06-15T07:36:33.897 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new 0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476 2 2024-06-15T07:36:33.931 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:33.933+0000 7f7ef80bc180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-15T07:36:34.094 INFO:teuthology.orchestra.run.smithi131.stdout:2 2024-06-15T07:36:34.111 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-06-15T07:36:34.111 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-06-15T07:36:34.124 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.125+0000 7f7ef80bc180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-15T07:36:34.125 INFO:tasks.ceph.c2.osd.0:Started 2024-06-15T07:36:34.126 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-06-15T07:36:34.126 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-06-15T07:36:34.127 INFO:tasks.ceph.c2.osd.1:Started 2024-06-15T07:36:34.128 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-06-15T07:36:34.128 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-06-15T07:36:34.130 INFO:tasks.ceph.c2.osd.2:Started 2024-06-15T07:36:34.130 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-06-15T07:36:34.307 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:34.307 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":5,"fsid":"cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee","created":"2024-06-15T07:36:33.207417+0000","modified":"2024-06-15T07:36:34.095083+0000","last_up_change":"0.000000","last_in_change":"2024-06-15T07:36:34.095083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b386e8-8bb6-43c9-bb3f-a7f27057c5bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a28a428-45ad-45ca-8ff7-3234b551efe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:34.322 INFO:tasks.ceph.ceph_manager.c2:[] 2024-06-15T07:36:34.323 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-15T07:36:34.625 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-06-15T07:36:34.696 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.697+0000 7f7ef80bc180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-15T07:36:34.728 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2024-06-15T07:36:34.729+0000 7fe37c1ff7c0 -1 Falling back to public interface 2024-06-15T07:36:34.753 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.753+0000 7f7ef80bc180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-15T07:36:34.755 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2024-06-15T07:36:34.757+0000 7f0a0a8747c0 -1 Falling back to public interface 2024-06-15T07:36:34.756 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2024-06-15T07:36:34.757+0000 7f8902b6c7c0 -1 Falling back to public interface 2024-06-15T07:36:34.807 INFO:teuthology.misc.health.smithi131.stdout: 2024-06-15T07:36:34.807 INFO:teuthology.misc.health.smithi131.stdout:{"epoch":5,"fsid":"cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee","created":"2024-06-15T07:36:33.207417+0000","modified":"2024-06-15T07:36:34.095083+0000","last_up_change":"0.000000","last_in_change":"2024-06-15T07:36:34.095083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b386e8-8bb6-43c9-bb3f-a7f27057c5bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a28a428-45ad-45ca-8ff7-3234b551efe9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:34.809 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.809+0000 7f7ef80bc180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-15T07:36:34.823 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-06-15T07:36:34.888 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.889+0000 7f7ef80bc180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-15T07:36:34.939 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:34.941+0000 7f7ef80bc180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-15T07:36:35.016 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:35.017+0000 7f7ef80bc180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-15T07:36:35.069 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:35.069+0000 7f7ef80bc180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-15T07:36:35.219 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:35.217+0000 7f7ef80bc180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-15T07:36:35.278 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:35.277+0000 7f7ef80bc180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-15T07:36:35.961 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-06-15T07:36:35.961 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-06-15T07:36:35.961 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr: from numpy import show_config as show_numpy_config 2024-06-15T07:36:36.026 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:36.025+0000 7f7ef80bc180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-15T07:36:36.189 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:36.189+0000 7f7ef80bc180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-15T07:36:36.360 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:36.361+0000 7f7ef80bc180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-15T07:36:36.430 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:36.429+0000 7f7ef80bc180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-15T07:36:36.578 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:36.577+0000 7f7ef80bc180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-15T07:36:37.198 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.197+0000 7f7ef80bc180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-15T07:36:37.274 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.273+0000 7f7ef80bc180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-15T07:36:37.456 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.457+0000 7f7ef80bc180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-15T07:36:37.490 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2024-06-15T07:36:37.489+0000 7f0a0a8747c0 -1 osd.2 0 log_to_monitors true 2024-06-15T07:36:37.535 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.537+0000 7f7ef80bc180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-15T07:36:37.772 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2024-06-15T07:36:37.773+0000 7fe37c1ff7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T07:36:37.793 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2024-06-15T07:36:37.793+0000 7f8902b6c7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T07:36:37.833 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.833+0000 7f7ef80bc180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-15T07:36:37.895 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:37.897+0000 7f7ef80bc180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-15T07:36:38.944 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2024-06-15T07:36:38.945+0000 7f7e9f672640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-15T07:36:39.093 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2024-06-15T07:36:39.093+0000 7f0a06819640 -1 osd.2 0 waiting for initial osdmap 2024-06-15T07:36:39.111 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2024-06-15T07:36:39.113+0000 7f09fe64c640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:40.624 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2024-06-15T07:36:40.625+0000 7f88feb11640 -1 osd.1 0 waiting for initial osdmap 2024-06-15T07:36:40.652 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2024-06-15T07:36:40.653+0000 7f88f6944640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:40.685 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2024-06-15T07:36:40.685+0000 7fe3781a4640 -1 osd.0 0 waiting for initial osdmap 2024-06-15T07:36:40.716 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2024-06-15T07:36:40.717+0000 7fe36ffd7640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T07:36:41.125 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-06-15T07:36:41.422 INFO:teuthology.misc.health.smithi131.stdout: 2024-06-15T07:36:41.422 INFO:teuthology.misc.health.smithi131.stdout:{"epoch":9,"fsid":"cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee","created":"2024-06-15T07:36:33.207417+0000","modified":"2024-06-15T07:36:40.995264+0000","last_up_change":"2024-06-15T07:36:40.995264+0000","last_in_change":"2024-06-15T07:36:34.095083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b386e8-8bb6-43c9-bb3f-a7f27057c5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":23137},{"type":"v1","addr":"172.21.15.131:6801","nonce":23137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":23137},{"type":"v1","addr":"172.21.15.131:6803","nonce":23137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":23137},{"type":"v1","addr":"172.21.15.131:6807","nonce":23137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":23137},{"type":"v1","addr":"172.21.15.131:6805","nonce":23137}]},"public_addr":"172.21.15.131:6801/23137","cluster_addr":"172.21.15.131:6803/23137","heartbeat_back_addr":"172.21.15.131:6807/23137","heartbeat_front_addr":"172.21.15.131:6805/23137","state":["exists","up"]},{"osd":1,"uuid":"7a28a428-45ad-45ca-8ff7-3234b551efe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":23147},{"type":"v1","addr":"172.21.15.131:6812","nonce":23147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":23147},{"type":"v1","addr":"172.21.15.131:6815","nonce":23147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6821","nonce":23147},{"type":"v1","addr":"172.21.15.131:6823","nonce":23147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6817","nonce":23147},{"type":"v1","addr":"172.21.15.131:6819","nonce":23147}]},"public_addr":"172.21.15.131:6812/23147","cluster_addr":"172.21.15.131:6815/23147","heartbeat_back_addr":"172.21.15.131:6823/23147","heartbeat_front_addr":"172.21.15.131:6819/23147","state":["exists","up"]},{"osd":2,"uuid":"0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":23138},{"type":"v1","addr":"172.21.15.131:6809","nonce":23138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":23138},{"type":"v1","addr":"172.21.15.131:6813","nonce":23138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":23138},{"type":"v1","addr":"172.21.15.131:6822","nonce":23138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":23138},{"type":"v1","addr":"172.21.15.131:6818","nonce":23138}]},"public_addr":"172.21.15.131:6809/23138","cluster_addr":"172.21.15.131:6813/23138","heartbeat_back_addr":"172.21.15.131:6822/23138","heartbeat_front_addr":"172.21.15.131:6818/23138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.506716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:41.440 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-06-15T07:36:41.440 INFO:tasks.ceph:Creating RBD pool 2024-06-15T07:36:41.440 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd pool create rbd 8 2024-06-15T07:36:42.574 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'rbd' created 2024-06-15T07:36:42.598 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-15T07:36:43.577 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-15T07:36:43.606 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-06-15T07:36:43.607 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-06-15T07:36:43.607 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-06-15T07:36:43.607 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-06-15T07:36:43.607 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-15T07:36:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:43.973 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":22961},{"type":"v1","addr":"172.21.15.131:6825","nonce":22961}]},"active_addr":"172.21.15.131:6825/22961","active_change":"2024-06-15T07:36:37.919821+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":3470102137}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":1110358054}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":1348900539}]}]} 2024-06-15T07:36:43.975 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-06-15T07:36:43.975 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-06-15T07:36:43.975 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-06-15T07:36:44.285 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:44.285 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":11,"fsid":"cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee","created":"2024-06-15T07:36:33.207417+0000","modified":"2024-06-15T07:36:43.572722+0000","last_up_change":"2024-06-15T07:36:40.995264+0000","last_in_change":"2024-06-15T07:36:34.095083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-06-15T07:36:41.771963+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-06-15T07:36:42.000109+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b386e8-8bb6-43c9-bb3f-a7f27057c5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":23137},{"type":"v1","addr":"172.21.15.131:6801","nonce":23137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":23137},{"type":"v1","addr":"172.21.15.131:6803","nonce":23137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":23137},{"type":"v1","addr":"172.21.15.131:6807","nonce":23137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":23137},{"type":"v1","addr":"172.21.15.131:6805","nonce":23137}]},"public_addr":"172.21.15.131:6801/23137","cluster_addr":"172.21.15.131:6803/23137","heartbeat_back_addr":"172.21.15.131:6807/23137","heartbeat_front_addr":"172.21.15.131:6805/23137","state":["exists","up"]},{"osd":1,"uuid":"7a28a428-45ad-45ca-8ff7-3234b551efe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":23147},{"type":"v1","addr":"172.21.15.131:6812","nonce":23147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":23147},{"type":"v1","addr":"172.21.15.131:6815","nonce":23147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6821","nonce":23147},{"type":"v1","addr":"172.21.15.131:6823","nonce":23147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6817","nonce":23147},{"type":"v1","addr":"172.21.15.131:6819","nonce":23147}]},"public_addr":"172.21.15.131:6812/23147","cluster_addr":"172.21.15.131:6815/23147","heartbeat_back_addr":"172.21.15.131:6823/23147","heartbeat_front_addr":"172.21.15.131:6819/23147","state":["exists","up"]},{"osd":2,"uuid":"0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":23138},{"type":"v1","addr":"172.21.15.131:6809","nonce":23138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":23138},{"type":"v1","addr":"172.21.15.131:6813","nonce":23138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":23138},{"type":"v1","addr":"172.21.15.131:6822","nonce":23138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":23138},{"type":"v1","addr":"172.21.15.131:6818","nonce":23138}]},"public_addr":"172.21.15.131:6809/23138","cluster_addr":"172.21.15.131:6813/23138","heartbeat_back_addr":"172.21.15.131:6822/23138","heartbeat_front_addr":"172.21.15.131:6818/23138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.769606+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.774110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.506716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:44.304 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-06-15T07:36:44.304 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-06-15T07:36:44.590 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:44.590 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":12,"fsid":"cbb0a994-f42e-43ee-ac63-bb7a6f6f85ee","created":"2024-06-15T07:36:33.207417+0000","modified":"2024-06-15T07:36:44.580803+0000","last_up_change":"2024-06-15T07:36:40.995264+0000","last_in_change":"2024-06-15T07:36:34.095083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-06-15T07:36:41.771963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-06-15T07:36:42.000109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b386e8-8bb6-43c9-bb3f-a7f27057c5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":23137},{"type":"v1","addr":"172.21.15.131:6801","nonce":23137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":23137},{"type":"v1","addr":"172.21.15.131:6803","nonce":23137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":23137},{"type":"v1","addr":"172.21.15.131:6807","nonce":23137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":23137},{"type":"v1","addr":"172.21.15.131:6805","nonce":23137}]},"public_addr":"172.21.15.131:6801/23137","cluster_addr":"172.21.15.131:6803/23137","heartbeat_back_addr":"172.21.15.131:6807/23137","heartbeat_front_addr":"172.21.15.131:6805/23137","state":["exists","up"]},{"osd":1,"uuid":"7a28a428-45ad-45ca-8ff7-3234b551efe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":23147},{"type":"v1","addr":"172.21.15.131:6812","nonce":23147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":23147},{"type":"v1","addr":"172.21.15.131:6815","nonce":23147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6821","nonce":23147},{"type":"v1","addr":"172.21.15.131:6823","nonce":23147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6817","nonce":23147},{"type":"v1","addr":"172.21.15.131:6819","nonce":23147}]},"public_addr":"172.21.15.131:6812/23147","cluster_addr":"172.21.15.131:6815/23147","heartbeat_back_addr":"172.21.15.131:6823/23147","heartbeat_front_addr":"172.21.15.131:6819/23147","state":["exists","up"]},{"osd":2,"uuid":"0d4aa9bc-0de2-4d7b-aff9-ad761b8ba476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":23138},{"type":"v1","addr":"172.21.15.131:6809","nonce":23138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6811","nonce":23138},{"type":"v1","addr":"172.21.15.131:6813","nonce":23138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":23138},{"type":"v1","addr":"172.21.15.131:6822","nonce":23138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":23138},{"type":"v1","addr":"172.21.15.131:6818","nonce":23138}]},"public_addr":"172.21.15.131:6809/23138","cluster_addr":"172.21.15.131:6813/23138","heartbeat_back_addr":"172.21.15.131:6822/23138","heartbeat_front_addr":"172.21.15.131:6818/23138","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.769606+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.774110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T07:36:38.506716+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T07:36:44.608 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-06-15T07:36:44.609 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-06-15T07:36:44.609 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-06-15T07:36:44.759 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2024-06-15T07:36:44.760 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-06-15T07:36:44.761 INFO:teuthology.orchestra.run.smithi131.stdout:38654705668 2024-06-15T07:36:44.761 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-06-15T07:36:44.764 INFO:teuthology.orchestra.run.smithi131.stdout:34359738370 2024-06-15T07:36:44.765 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-06-15T07:36:45.080 INFO:teuthology.orchestra.run.smithi131.stdout:34359738369 2024-06-15T07:36:45.097 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-06-15T07:36:45.120 INFO:teuthology.orchestra.run.smithi131.stdout:38654705666 2024-06-15T07:36:45.137 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2024-06-15T07:36:45.160 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2024-06-15T07:36:45.177 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.1 2024-06-15T07:36:46.098 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-06-15T07:36:46.138 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-06-15T07:36:46.178 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-06-15T07:36:46.394 INFO:teuthology.orchestra.run.smithi131.stdout:34359738369 2024-06-15T07:36:46.411 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.2 2024-06-15T07:36:46.432 INFO:teuthology.orchestra.run.smithi131.stdout:38654705666 2024-06-15T07:36:46.448 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2024-06-15T07:36:46.468 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2024-06-15T07:36:46.485 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.1 2024-06-15T07:36:47.412 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-06-15T07:36:47.449 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-06-15T07:36:47.485 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-06-15T07:36:47.719 INFO:teuthology.orchestra.run.smithi131.stdout:34359738371 2024-06-15T07:36:47.742 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738371 for osd.2 2024-06-15T07:36:47.743 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:47.750 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2024-06-15T07:36:47.767 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.0 2024-06-15T07:36:47.767 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:47.817 INFO:teuthology.orchestra.run.smithi131.stdout:38654705668 2024-06-15T07:36:47.834 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.1 2024-06-15T07:36:47.834 DEBUG:teuthology.parallel:result is None 2024-06-15T07:36:47.834 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-06-15T07:36:47.834 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-06-15T07:36:48.115 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:48.115 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2024-06-15T07:36:48.133 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-06-15T07:36:47.930227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82528,"kb_used_data":1460,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490272,"statfs":{"total":322122547200,"available":322038038528,"internally_reserved":0,"allocated":1495040,"data_stored":1262119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000511"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742640+0000","last_change":"2024-06-15T07:36:43.588878+0000","last_active":"2024-06-15T07:36:44.742640+0000","last_peered":"2024-06-15T07:36:44.742640+0000","last_clean":"2024-06-15T07:36:44.742640+0000","last_became_active":"2024-06-15T07:36:43.588656+0000","last_became_peered":"2024-06-15T07:36:43.588656+0000","last_unstale":"2024-06-15T07:36:44.742640+0000","last_undegraded":"2024-06-15T07:36:44.742640+0000","last_fullsized":"2024-06-15T07:36:44.742640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T17:11:45.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742629+0000","last_change":"2024-06-15T07:36:43.583791+0000","last_active":"2024-06-15T07:36:44.742629+0000","last_peered":"2024-06-15T07:36:44.742629+0000","last_clean":"2024-06-15T07:36:44.742629+0000","last_became_active":"2024-06-15T07:36:43.583544+0000","last_became_peered":"2024-06-15T07:36:43.583544+0000","last_unstale":"2024-06-15T07:36:44.742629+0000","last_undegraded":"2024-06-15T07:36:44.742629+0000","last_fullsized":"2024-06-15T07:36:44.742629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:04:43.993658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584644+0000","last_change":"2024-06-15T07:36:43.593688+0000","last_active":"2024-06-15T07:36:44.584644+0000","last_peered":"2024-06-15T07:36:44.584644+0000","last_clean":"2024-06-15T07:36:44.584644+0000","last_became_active":"2024-06-15T07:36:43.593409+0000","last_became_peered":"2024-06-15T07:36:43.593409+0000","last_unstale":"2024-06-15T07:36:44.584644+0000","last_undegraded":"2024-06-15T07:36:44.584644+0000","last_fullsized":"2024-06-15T07:36:44.584644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:15.428290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742618+0000","last_change":"2024-06-15T07:36:43.584077+0000","last_active":"2024-06-15T07:36:44.742618+0000","last_peered":"2024-06-15T07:36:44.742618+0000","last_clean":"2024-06-15T07:36:44.742618+0000","last_became_active":"2024-06-15T07:36:43.583732+0000","last_became_peered":"2024-06-15T07:36:43.583732+0000","last_unstale":"2024-06-15T07:36:44.742618+0000","last_undegraded":"2024-06-15T07:36:44.742618+0000","last_fullsized":"2024-06-15T07:36:44.742618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:23.514930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584509+0000","last_change":"2024-06-15T07:36:43.589263+0000","last_active":"2024-06-15T07:36:44.584509+0000","last_peered":"2024-06-15T07:36:44.584509+0000","last_clean":"2024-06-15T07:36:44.584509+0000","last_became_active":"2024-06-15T07:36:43.588621+0000","last_became_peered":"2024-06-15T07:36:43.588621+0000","last_unstale":"2024-06-15T07:36:44.584509+0000","last_undegraded":"2024-06-15T07:36:44.584509+0000","last_fullsized":"2024-06-15T07:36:44.584509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T13:15:36.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742607+0000","last_change":"2024-06-15T07:36:43.588280+0000","last_active":"2024-06-15T07:36:44.742607+0000","last_peered":"2024-06-15T07:36:44.742607+0000","last_clean":"2024-06-15T07:36:44.742607+0000","last_became_active":"2024-06-15T07:36:43.588028+0000","last_became_peered":"2024-06-15T07:36:43.588028+0000","last_unstale":"2024-06-15T07:36:44.742607+0000","last_undegraded":"2024-06-15T07:36:44.742607+0000","last_fullsized":"2024-06-15T07:36:44.742607+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:32:56.124697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:43.587969+0000","last_change":"2024-06-15T07:36:43.585962+0000","last_active":"2024-06-15T07:36:43.587969+0000","last_peered":"2024-06-15T07:36:43.587969+0000","last_clean":"2024-06-15T07:36:43.587969+0000","last_became_active":"2024-06-15T07:36:43.585703+0000","last_became_peered":"2024-06-15T07:36:43.585703+0000","last_unstale":"2024-06-15T07:36:43.587969+0000","last_undegraded":"2024-06-15T07:36:43.587969+0000","last_fullsized":"2024-06-15T07:36:43.587969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T13:37:05.197642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584661+0000","last_change":"2024-06-15T07:36:43.589057+0000","last_active":"2024-06-15T07:36:44.584661+0000","last_peered":"2024-06-15T07:36:44.584661+0000","last_clean":"2024-06-15T07:36:44.584661+0000","last_became_active":"2024-06-15T07:36:43.588739+0000","last_became_peered":"2024-06-15T07:36:43.588739+0000","last_unstale":"2024-06-15T07:36:44.584661+0000","last_undegraded":"2024-06-15T07:36:44.584661+0000","last_fullsized":"2024-06-15T07:36:44.584661+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:26.523399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742591+0000","last_change":"2024-06-15T07:36:43.590811+0000","last_active":"2024-06-15T07:36:44.742591+0000","last_peered":"2024-06-15T07:36:44.742591+0000","last_clean":"2024-06-15T07:36:44.742591+0000","last_became_active":"2024-06-15T07:36:43.590499+0000","last_became_peered":"2024-06-15T07:36:43.590499+0000","last_unstale":"2024-06-15T07:36:44.742591+0000","last_undegraded":"2024-06-15T07:36:44.742591+0000","last_fullsized":"2024-06-15T07:36:44.742591+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:53:20.873930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":700416,"data_stored":619132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T07:36:48.134 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-06-15T07:36:48.407 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:48.408 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2024-06-15T07:36:48.424 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-06-15T07:36:47.930227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82528,"kb_used_data":1460,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490272,"statfs":{"total":322122547200,"available":322038038528,"internally_reserved":0,"allocated":1495040,"data_stored":1262119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000511"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742640+0000","last_change":"2024-06-15T07:36:43.588878+0000","last_active":"2024-06-15T07:36:44.742640+0000","last_peered":"2024-06-15T07:36:44.742640+0000","last_clean":"2024-06-15T07:36:44.742640+0000","last_became_active":"2024-06-15T07:36:43.588656+0000","last_became_peered":"2024-06-15T07:36:43.588656+0000","last_unstale":"2024-06-15T07:36:44.742640+0000","last_undegraded":"2024-06-15T07:36:44.742640+0000","last_fullsized":"2024-06-15T07:36:44.742640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T17:11:45.434947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742629+0000","last_change":"2024-06-15T07:36:43.583791+0000","last_active":"2024-06-15T07:36:44.742629+0000","last_peered":"2024-06-15T07:36:44.742629+0000","last_clean":"2024-06-15T07:36:44.742629+0000","last_became_active":"2024-06-15T07:36:43.583544+0000","last_became_peered":"2024-06-15T07:36:43.583544+0000","last_unstale":"2024-06-15T07:36:44.742629+0000","last_undegraded":"2024-06-15T07:36:44.742629+0000","last_fullsized":"2024-06-15T07:36:44.742629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:04:43.993658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584644+0000","last_change":"2024-06-15T07:36:43.593688+0000","last_active":"2024-06-15T07:36:44.584644+0000","last_peered":"2024-06-15T07:36:44.584644+0000","last_clean":"2024-06-15T07:36:44.584644+0000","last_became_active":"2024-06-15T07:36:43.593409+0000","last_became_peered":"2024-06-15T07:36:43.593409+0000","last_unstale":"2024-06-15T07:36:44.584644+0000","last_undegraded":"2024-06-15T07:36:44.584644+0000","last_fullsized":"2024-06-15T07:36:44.584644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:15.428290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742618+0000","last_change":"2024-06-15T07:36:43.584077+0000","last_active":"2024-06-15T07:36:44.742618+0000","last_peered":"2024-06-15T07:36:44.742618+0000","last_clean":"2024-06-15T07:36:44.742618+0000","last_became_active":"2024-06-15T07:36:43.583732+0000","last_became_peered":"2024-06-15T07:36:43.583732+0000","last_unstale":"2024-06-15T07:36:44.742618+0000","last_undegraded":"2024-06-15T07:36:44.742618+0000","last_fullsized":"2024-06-15T07:36:44.742618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:23.514930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":137,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584509+0000","last_change":"2024-06-15T07:36:43.589263+0000","last_active":"2024-06-15T07:36:44.584509+0000","last_peered":"2024-06-15T07:36:44.584509+0000","last_clean":"2024-06-15T07:36:44.584509+0000","last_became_active":"2024-06-15T07:36:43.588621+0000","last_became_peered":"2024-06-15T07:36:43.588621+0000","last_unstale":"2024-06-15T07:36:44.584509+0000","last_undegraded":"2024-06-15T07:36:44.584509+0000","last_fullsized":"2024-06-15T07:36:44.584509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T13:15:36.558584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742607+0000","last_change":"2024-06-15T07:36:43.588280+0000","last_active":"2024-06-15T07:36:44.742607+0000","last_peered":"2024-06-15T07:36:44.742607+0000","last_clean":"2024-06-15T07:36:44.742607+0000","last_became_active":"2024-06-15T07:36:43.588028+0000","last_became_peered":"2024-06-15T07:36:43.588028+0000","last_unstale":"2024-06-15T07:36:44.742607+0000","last_undegraded":"2024-06-15T07:36:44.742607+0000","last_fullsized":"2024-06-15T07:36:44.742607+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:32:56.124697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:43.587969+0000","last_change":"2024-06-15T07:36:43.585962+0000","last_active":"2024-06-15T07:36:43.587969+0000","last_peered":"2024-06-15T07:36:43.587969+0000","last_clean":"2024-06-15T07:36:43.587969+0000","last_became_active":"2024-06-15T07:36:43.585703+0000","last_became_peered":"2024-06-15T07:36:43.585703+0000","last_unstale":"2024-06-15T07:36:43.587969+0000","last_undegraded":"2024-06-15T07:36:43.587969+0000","last_fullsized":"2024-06-15T07:36:43.587969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T13:37:05.197642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.584661+0000","last_change":"2024-06-15T07:36:43.589057+0000","last_active":"2024-06-15T07:36:44.584661+0000","last_peered":"2024-06-15T07:36:44.584661+0000","last_clean":"2024-06-15T07:36:44.584661+0000","last_became_active":"2024-06-15T07:36:43.588739+0000","last_became_peered":"2024-06-15T07:36:43.588739+0000","last_unstale":"2024-06-15T07:36:44.584661+0000","last_undegraded":"2024-06-15T07:36:44.584661+0000","last_fullsized":"2024-06-15T07:36:44.584661+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:26.523399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-06-15T07:36:44.742591+0000","last_change":"2024-06-15T07:36:43.590811+0000","last_active":"2024-06-15T07:36:44.742591+0000","last_peered":"2024-06-15T07:36:44.742591+0000","last_clean":"2024-06-15T07:36:44.742591+0000","last_became_active":"2024-06-15T07:36:43.590499+0000","last_became_peered":"2024-06-15T07:36:43.590499+0000","last_unstale":"2024-06-15T07:36:44.742591+0000","last_undegraded":"2024-06-15T07:36:44.742591+0000","last_fullsized":"2024-06-15T07:36:44.742591+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T07:36:42.567399+0000","last_clean_scrub_stamp":"2024-06-15T07:36:42.567399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:53:20.873930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":24916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27704,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":700416,"data_stored":619132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T07:36:48.424 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-06-15T07:36:48.425 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-06-15T07:36:48.425 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-06-15T07:36:48.425 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-06-15T07:36:48.721 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-06-15T07:36:48.721 INFO:teuthology.orchestra.run.smithi131.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-15T07:36:48.738 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-06-15T07:36:48.739 INFO:teuthology.run_tasks:Running task rgw... 2024-06-15T07:36:48.756 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-06-15T07:36:48.757 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-06-15T07:36:48.757 INFO:tasks.rgw:Creating data pools 2024-06-15T07:36:48.757 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-06-15T07:36:48.758 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-06-15T07:36:49.796 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.data' created 2024-06-15T07:36:49.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-06-15T07:36:50.818 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-06-15T07:36:50.866 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-06-15T07:36:51.818 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.index' created 2024-06-15T07:36:51.855 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-06-15T07:36:52.829 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-06-15T07:36:52.880 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-06-15T07:36:52.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-06-15T07:36:53.180 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.data' already exists 2024-06-15T07:36:53.199 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-06-15T07:36:53.828 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-06-15T07:36:53.850 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-06-15T07:36:54.161 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.index' already exists 2024-06-15T07:36:54.180 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-06-15T07:36:54.833 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-06-15T07:36:54.857 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-06-15T07:36:54.857 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-06-15T07:36:55.997 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.data' created 2024-06-15T07:36:56.028 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-06-15T07:36:57.008 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-06-15T07:36:57.058 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-06-15T07:36:58.012 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.index' created 2024-06-15T07:36:58.044 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-06-15T07:36:59.017 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-06-15T07:36:59.062 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-06-15T07:36:59.062 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-06-15T07:36:59.346 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.data' already exists 2024-06-15T07:36:59.363 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-06-15T07:37:00.029 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-06-15T07:37:00.051 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-06-15T07:37:00.335 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.index' already exists 2024-06-15T07:37:00.352 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-06-15T07:37:01.029 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-06-15T07:37:01.054 DEBUG:tasks.rgw:Pools created 2024-06-15T07:37:01.054 INFO:tasks.rgw:Configuring compression type = random 2024-06-15T07:37:01.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-06-15T07:37:01.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-06-15T07:37:01.055 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-06-15T07:37:01.120 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:01.120 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:01.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:01.131+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 realm 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-06-15T07:37:02.873 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.871+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.887+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.889 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.887+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.887+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.887+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.891+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.891+0000 7faf46ea9a40 10 Creating default zonegroup 2024-06-15T07:37:02.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.891+0000 7faf46ea9a40 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-06-15T07:37:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.911+0000 7faf46ea9a40 10 WARNING: driver->list_zones() returned r=-2 2024-06-15T07:37:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.911+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.911+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 10 could not read realm id: (2) No such file or directory 2024-06-15T07:37:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 10 WARNING: failed to set zone as default, r=-22 2024-06-15T07:37:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.915+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.919+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.919+0000 7faf46ea9a40 20 rados_obj.operate() r=0 bl.length=874 2024-06-15T07:37:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.919+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados_obj.operate() r=0 bl.length=420 2024-06-15T07:37:02.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 10 Using default name default 2024-06-15T07:37:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:02.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados_obj.operate() r=0 bl.length=874 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 zone default found 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 4 Realm: () 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 4 ZoneGroup: default (5cbe1ede-aad4-42d8-aa46-00abb2f80661) 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 4 Zone: default (15422203-6995-423b-83d7-fe079ff1fcb5) 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 10 cannot find current period zonegroup using local zonegroup configuration 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 zonegroup default 2024-06-15T07:37:02.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:02.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:02.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:02.923+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:04.887+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:04.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:04.887+0000 7faf46ea9a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:04.891+0000 7faf46ea9a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:04.891+0000 7faf46ea9a40 20 started sync module instance, tier type = 2024-06-15T07:37:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:04.891+0000 7faf46ea9a40 20 started zone id=15422203-6995-423b-83d7-fe079ff1fcb5 (name=default) with tier type = 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=0 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=1 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=2 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=3 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=4 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=5 2024-06-15T07:37:06.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=6 2024-06-15T07:37:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 add_watcher() i=7 2024-06-15T07:37:06.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 2 all 8 watchers are set, enabling cache 2024-06-15T07:37:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-06-15T07:37:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-06-15T07:37:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:06.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.975+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-06-15T07:37:06.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-06-15T07:37:06.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:06.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-06-15T07:37:06.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-06-15T07:37:06.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.979+0000 7faf46ea9a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:06.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.987+0000 7faf2c634640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:06.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.987+0000 7faf46ea9a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:06.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:06.987+0000 7faf46ea9a40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:09.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.911+0000 7faf46ea9a40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-06-15T07:37:09.916 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.911+0000 7faf46ea9a40 5 note: GC not initialized 2024-06-15T07:37:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.923+0000 7faf27e2b640 20 reqs_thread_entry: start 2024-06-15T07:37:09.928 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.923+0000 7faf46ea9a40 20 init_complete bucket index max shards: 11 2024-06-15T07:37:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.923+0000 7faf46ea9a40 10 rgw notify: Started notification manager with: 1 workers 2024-06-15T07:37:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.923+0000 7faf25e27640 20 rgw notify: INFO: next queues processing will happen at: Sat Jun 15 07:37:40 2024 2024-06-15T07:37:09.929 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-06-15T07:37:09.929 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.923+0000 7faf25605640 20 reqs_thread_entry: start 2024-06-15T07:37:09.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.955+0000 7faf2c634640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:09.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.959+0000 7faf46ea9a40 20 remove_watcher() i=0 2024-06-15T07:37:09.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.959+0000 7faf46ea9a40 2 removed watcher, disabling cache 2024-06-15T07:37:09.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.959+0000 7faf46ea9a40 20 remove_watcher() i=1 2024-06-15T07:37:09.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.963+0000 7faf46ea9a40 20 remove_watcher() i=2 2024-06-15T07:37:09.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.967+0000 7faf46ea9a40 20 remove_watcher() i=3 2024-06-15T07:37:09.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.967+0000 7faf46ea9a40 20 remove_watcher() i=4 2024-06-15T07:37:09.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.971+0000 7faf46ea9a40 20 remove_watcher() i=5 2024-06-15T07:37:09.974 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.971+0000 7faf46ea9a40 20 remove_watcher() i=6 2024-06-15T07:37:09.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:09.975+0000 7faf46ea9a40 20 remove_watcher() i=7 2024-06-15T07:37:09.981 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-06-15T07:37:09.984 DEBUG:tasks.util.rgw: json result: [] 2024-06-15T07:37:09.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:09.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:09.985 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-06-15T07:37:10.029 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:10.030 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.039+0000 7fbcb2387a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:10.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.039+0000 7fbcb2387a40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:10.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.043+0000 7fbc9430b640 20 reqs_thread_entry: start 2024-06-15T07:37:10.055 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:10.056 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "15422203-6995-423b-83d7-fe079ff1fcb5", 2024-06-15T07:37:10.056 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default", 2024-06-15T07:37:10.056 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "default.rgw.meta:root", 2024-06-15T07:37:10.056 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "default.rgw.control", 2024-06-15T07:37:10.056 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "default.rgw.log:gc", 2024-06-15T07:37:10.057 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "default.rgw.log:lc", 2024-06-15T07:37:10.057 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "default.rgw.log", 2024-06-15T07:37:10.057 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-06-15T07:37:10.058 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-06-15T07:37:10.058 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "default.rgw.meta:roles", 2024-06-15T07:37:10.058 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-06-15T07:37:10.058 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-06-15T07:37:10.058 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "default.rgw.otp", 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "", 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "" 2024-06-15T07:37:10.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:10.060 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:10.060 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:10.060 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "default.rgw.buckets.index", 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "default.rgw.buckets.data", 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: "compression_type": "random" 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "", 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "default.rgw.log:notif" 2024-06-15T07:37:10.062 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:10.065 DEBUG:tasks.util.rgw: json result: {'id': '15422203-6995-423b-83d7-fe079ff1fcb5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-06-15T07:37:10.065 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-06-15T07:37:10.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-06-15T07:37:10.067 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-06-15T07:37:10.146 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:10.146 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:10.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.155+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.155+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.155+0000 7fbd02821a40 20 realm 2024-06-15T07:37:10.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.155+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-06-15T07:37:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 rados_obj.operate() r=0 bl.length=890 2024-06-15T07:37:10.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.159+0000 7fbd02821a40 20 searching for the correct realm 2024-06-15T07:37:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 RGWRados::pool_iterate: got zonegroup_info.5cbe1ede-aad4-42d8-aa46-00abb2f80661 2024-06-15T07:37:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 RGWRados::pool_iterate: got zone_info.15422203-6995-423b-83d7-fe079ff1fcb5 2024-06-15T07:37:10.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 RGWRados::pool_iterate: got zone_names.default 2024-06-15T07:37:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-06-15T07:37:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=0 bl.length=420 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 zone default found 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 4 Realm: () 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 4 ZoneGroup: default (5cbe1ede-aad4-42d8-aa46-00abb2f80661) 2024-06-15T07:37:10.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 4 Zone: default (15422203-6995-423b-83d7-fe079ff1fcb5) 2024-06-15T07:37:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 10 cannot find current period zonegroup using local zonegroup configuration 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 zonegroup default 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 started sync module instance, tier type = 2024-06-15T07:37:10.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.167+0000 7fbd02821a40 20 started zone id=15422203-6995-423b-83d7-fe079ff1fcb5 (name=default) with tier type = 2024-06-15T07:37:10.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=0 2024-06-15T07:37:10.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=1 2024-06-15T07:37:10.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=2 2024-06-15T07:37:10.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=3 2024-06-15T07:37:10.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=4 2024-06-15T07:37:10.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=5 2024-06-15T07:37:10.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.199+0000 7fbd02821a40 20 add_watcher() i=6 2024-06-15T07:37:10.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.203+0000 7fbd02821a40 20 add_watcher() i=7 2024-06-15T07:37:10.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.203+0000 7fbd02821a40 2 all 8 watchers are set, enabling cache 2024-06-15T07:37:10.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.203+0000 7fbce7fac640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:10.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.203+0000 7fbd02821a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:10.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.203+0000 7fbd02821a40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:10.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.207+0000 7fbd02821a40 5 note: GC not initialized 2024-06-15T07:37:10.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.207+0000 7fbce37a3640 20 reqs_thread_entry: start 2024-06-15T07:37:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.211+0000 7fbd02821a40 20 init_complete bucket index max shards: 11 2024-06-15T07:37:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.211+0000 7fbd02821a40 10 rgw notify: Started notification manager with: 1 workers 2024-06-15T07:37:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.211+0000 7fbce179f640 20 rgw notify: INFO: next queues processing will happen at: Sat Jun 15 07:37:40 2024 2024-06-15T07:37:10.213 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-06-15T07:37:10.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.211+0000 7fbce0f7d640 20 reqs_thread_entry: start 2024-06-15T07:37:10.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.219+0000 7fbce7fac640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:10.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.223+0000 7fbd02821a40 20 remove_watcher() i=0 2024-06-15T07:37:10.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.223+0000 7fbd02821a40 2 removed watcher, disabling cache 2024-06-15T07:37:10.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.223+0000 7fbd02821a40 20 remove_watcher() i=1 2024-06-15T07:37:10.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.223+0000 7fbd02821a40 20 remove_watcher() i=2 2024-06-15T07:37:10.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.227+0000 7fbd02821a40 20 remove_watcher() i=3 2024-06-15T07:37:10.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.231+0000 7fbd02821a40 20 remove_watcher() i=4 2024-06-15T07:37:10.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.231+0000 7fbd02821a40 20 remove_watcher() i=5 2024-06-15T07:37:10.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.235+0000 7fbd02821a40 20 remove_watcher() i=6 2024-06-15T07:37:10.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.235+0000 7fbd02821a40 20 remove_watcher() i=7 2024-06-15T07:37:10.242 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2024-06-15T07:37:10.245 DEBUG:tasks.util.rgw: json result: [] 2024-06-15T07:37:10.245 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:10.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:10.245 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-06-15T07:37:10.324 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:10.324 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:10.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.335+0000 7fac51e14a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:10.337 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.335+0000 7fac51e14a40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:10.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:10.339+0000 7fac33d98640 20 reqs_thread_entry: start 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "15422203-6995-423b-83d7-fe079ff1fcb5", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "default.rgw.meta:root", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "default.rgw.control", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "default.rgw.log:gc", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "default.rgw.log:lc", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "default.rgw.log", 2024-06-15T07:37:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "default.rgw.meta:roles", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "default.rgw.otp", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "", 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "" 2024-06-15T07:37:10.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "default.rgw.buckets.index", 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "default.rgw.buckets.data", 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "compression_type": "random" 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:10.351 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "", 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "default.rgw.log:notif" 2024-06-15T07:37:10.352 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:10.353 DEBUG:tasks.util.rgw: json result: {'id': '15422203-6995-423b-83d7-fe079ff1fcb5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-06-15T07:37:10.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-06-15T07:37:10.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-06-15T07:37:10.353 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-06-15T07:37:10.410 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:10.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:10.426 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:10.429+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.093+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.093+0000 7f64951f4a40 20 realm 2024-06-15T07:37:12.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.093+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.096 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.097+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.097+0000 7f64951f4a40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-06-15T07:37:12.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.097+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 10 Creating default zonegroup 2024-06-15T07:37:12.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.101+0000 7f64951f4a40 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-06-15T07:37:12.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.117+0000 7f64951f4a40 10 WARNING: driver->list_zones() returned r=-2 2024-06-15T07:37:12.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.117+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.117+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 10 could not read realm id: (2) No such file or directory 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 10 WARNING: failed to set zone as default, r=-22 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.121+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 20 rados_obj.operate() r=0 bl.length=874 2024-06-15T07:37:12.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 20 rados_obj.operate() r=0 bl.length=420 2024-06-15T07:37:12.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 10 Using default name default 2024-06-15T07:37:12.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.125+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:12.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados_obj.operate() r=0 bl.length=874 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 zone default found 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 4 Realm: () 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 4 ZoneGroup: default (4c712d9c-d26a-4064-9f50-a8c0ec9d624d) 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 4 Zone: default (9ba29813-939b-4814-b0a1-c7153b87e285) 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 10 cannot find current period zonegroup using local zonegroup configuration 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 zonegroup default 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:12.129+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:14.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:14.125+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:14.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:14.125+0000 7f64951f4a40 20 rados->read ofs=0 len=0 2024-06-15T07:37:14.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:14.133+0000 7f64951f4a40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:14.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:14.133+0000 7f64951f4a40 20 started sync module instance, tier type = 2024-06-15T07:37:14.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:14.133+0000 7f64951f4a40 20 started zone id=9ba29813-939b-4814-b0a1-c7153b87e285 (name=default) with tier type = 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=0 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=1 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=2 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=3 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=4 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=5 2024-06-15T07:37:16.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.177+0000 7f64951f4a40 20 add_watcher() i=6 2024-06-15T07:37:16.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 add_watcher() i=7 2024-06-15T07:37:16.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 2 all 8 watchers are set, enabling cache 2024-06-15T07:37:16.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-06-15T07:37:16.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-06-15T07:37:16.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-06-15T07:37:16.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:16.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-06-15T07:37:16.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-06-15T07:37:16.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:16.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-06-15T07:37:16.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.181+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-06-15T07:37:16.184 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.185+0000 7f64951f4a40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-06-15T07:37:16.190 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.193+0000 7f647a97f640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:16.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.193+0000 7f64951f4a40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:16.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:16.193+0000 7f64951f4a40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:19.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.113+0000 7f64951f4a40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-06-15T07:37:19.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.113+0000 7f64951f4a40 5 note: GC not initialized 2024-06-15T07:37:19.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.129+0000 7f6476176640 20 reqs_thread_entry: start 2024-06-15T07:37:19.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.129+0000 7f64951f4a40 20 init_complete bucket index max shards: 11 2024-06-15T07:37:19.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.129+0000 7f64951f4a40 10 rgw notify: Started notification manager with: 1 workers 2024-06-15T07:37:19.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.129+0000 7f6474172640 20 rgw notify: INFO: next queues processing will happen at: Sat Jun 15 07:37:49 2024 2024-06-15T07:37:19.130 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-06-15T07:37:19.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.129+0000 7f6473950640 20 reqs_thread_entry: start 2024-06-15T07:37:19.155 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.157+0000 7f647a97f640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:19.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.157+0000 7f64951f4a40 20 remove_watcher() i=0 2024-06-15T07:37:19.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.157+0000 7f64951f4a40 2 removed watcher, disabling cache 2024-06-15T07:37:19.159 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.161+0000 7f64951f4a40 20 remove_watcher() i=1 2024-06-15T07:37:19.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.161+0000 7f64951f4a40 20 remove_watcher() i=2 2024-06-15T07:37:19.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.161+0000 7f64951f4a40 20 remove_watcher() i=3 2024-06-15T07:37:19.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.165+0000 7f64951f4a40 20 remove_watcher() i=4 2024-06-15T07:37:19.164 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.165+0000 7f64951f4a40 20 remove_watcher() i=5 2024-06-15T07:37:19.165 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.165+0000 7f64951f4a40 20 remove_watcher() i=6 2024-06-15T07:37:19.166 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.169+0000 7f64951f4a40 20 remove_watcher() i=7 2024-06-15T07:37:19.170 INFO:teuthology.orchestra.run.smithi131.stdout:[] 2024-06-15T07:37:19.172 DEBUG:tasks.util.rgw: json result: [] 2024-06-15T07:37:19.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:19.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:19.173 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-06-15T07:37:19.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:19.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:19.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.229+0000 7fd8e0f3ca40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:19.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.229+0000 7fd8e0f3ca40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:19.230 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.229+0000 7fd8c2ec0640 20 reqs_thread_entry: start 2024-06-15T07:37:19.238 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:19.239 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "9ba29813-939b-4814-b0a1-c7153b87e285", 2024-06-15T07:37:19.239 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default", 2024-06-15T07:37:19.239 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "default.rgw.meta:root", 2024-06-15T07:37:19.239 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "default.rgw.control", 2024-06-15T07:37:19.239 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "default.rgw.log:gc", 2024-06-15T07:37:19.241 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "default.rgw.log:lc", 2024-06-15T07:37:19.241 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "default.rgw.log", 2024-06-15T07:37:19.241 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-06-15T07:37:19.241 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-06-15T07:37:19.241 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "default.rgw.meta:roles", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "default.rgw.otp", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "", 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "" 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:19.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "default.rgw.buckets.index", 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "default.rgw.buckets.data", 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.243 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:19.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "", 2024-06-15T07:37:19.244 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "default.rgw.log:notif" 2024-06-15T07:37:19.244 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:19.245 DEBUG:tasks.util.rgw: json result: {'id': '9ba29813-939b-4814-b0a1-c7153b87e285', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-06-15T07:37:19.245 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-06-15T07:37:19.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-06-15T07:37:19.245 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-06-15T07:37:19.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:19.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:19.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 realm 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.334 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:19.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.333+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.337+0000 7f1fd750fa40 20 rados_obj.operate() r=0 bl.length=890 2024-06-15T07:37:19.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.337+0000 7f1fd750fa40 20 searching for the correct realm 2024-06-15T07:37:19.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 RGWRados::pool_iterate: got zone_info.9ba29813-939b-4814-b0a1-c7153b87e285 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 RGWRados::pool_iterate: got zonegroup_info.4c712d9c-d26a-4064-9f50-a8c0ec9d624d 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 RGWRados::pool_iterate: got zone_names.default 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=0 bl.length=46 2024-06-15T07:37:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=0 bl.length=420 2024-06-15T07:37:19.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 zone default found 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 4 Realm: () 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 4 ZoneGroup: default (4c712d9c-d26a-4064-9f50-a8c0ec9d624d) 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 4 Zone: default (9ba29813-939b-4814-b0a1-c7153b87e285) 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 10 cannot find current period zonegroup using local zonegroup configuration 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 zonegroup default 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados->read ofs=0 len=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 rados_obj.operate() r=-2 bl.length=0 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 started sync module instance, tier type = 2024-06-15T07:37:19.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.345+0000 7f1fd750fa40 20 started zone id=9ba29813-939b-4814-b0a1-c7153b87e285 (name=default) with tier type = 2024-06-15T07:37:19.356 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=0 2024-06-15T07:37:19.356 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=1 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=2 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=3 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=4 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=5 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=6 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 20 add_watcher() i=7 2024-06-15T07:37:19.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.357+0000 7f1fd750fa40 2 all 8 watchers are set, enabling cache 2024-06-15T07:37:19.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.361+0000 7f1fbcc9a640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:19.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.361+0000 7f1fd750fa40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:19.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.361+0000 7f1fd750fa40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:19.363 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fd750fa40 5 note: GC not initialized 2024-06-15T07:37:19.363 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fb8491640 20 reqs_thread_entry: start 2024-06-15T07:37:19.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fd750fa40 20 init_complete bucket index max shards: 11 2024-06-15T07:37:19.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fd750fa40 10 rgw notify: Started notification manager with: 1 workers 2024-06-15T07:37:19.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fb648d640 20 rgw notify: INFO: next queues processing will happen at: Sat Jun 15 07:37:49 2024 2024-06-15T07:37:19.364 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-06-15T07:37:19.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.365+0000 7f1fb5c6b640 20 reqs_thread_entry: start 2024-06-15T07:37:19.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.373+0000 7f1fbcc9a640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-06-15T07:37:19.375 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.377+0000 7f1fd750fa40 20 remove_watcher() i=0 2024-06-15T07:37:19.375 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.377+0000 7f1fd750fa40 2 removed watcher, disabling cache 2024-06-15T07:37:19.376 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.377+0000 7f1fd750fa40 20 remove_watcher() i=1 2024-06-15T07:37:19.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.377+0000 7f1fd750fa40 20 remove_watcher() i=2 2024-06-15T07:37:19.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.377+0000 7f1fd750fa40 20 remove_watcher() i=3 2024-06-15T07:37:19.378 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.381+0000 7f1fd750fa40 20 remove_watcher() i=4 2024-06-15T07:37:19.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.381+0000 7f1fd750fa40 20 remove_watcher() i=5 2024-06-15T07:37:19.382 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.381+0000 7f1fd750fa40 20 remove_watcher() i=6 2024-06-15T07:37:19.383 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.385+0000 7f1fd750fa40 20 remove_watcher() i=7 2024-06-15T07:37:19.386 INFO:teuthology.orchestra.run.smithi131.stdout:[] 2024-06-15T07:37:19.388 DEBUG:tasks.util.rgw: json result: [] 2024-06-15T07:37:19.388 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:19.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-06-15T07:37:19.388 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-06-15T07:37:19.470 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:19.470 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:19.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.485+0000 7f26d4a5ba40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-06-15T07:37:19.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.485+0000 7f26d4a5ba40 20 check_secure_mon_conn(): mode 1 is insecure 2024-06-15T07:37:19.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:19.489+0000 7f26b69df640 20 reqs_thread_entry: start 2024-06-15T07:37:19.497 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:19.497 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "9ba29813-939b-4814-b0a1-c7153b87e285", 2024-06-15T07:37:19.497 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "default.rgw.meta:root", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "default.rgw.control", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "default.rgw.log:gc", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "default.rgw.log:lc", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "default.rgw.log", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "default.rgw.meta:roles", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-06-15T07:37:19.498 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "default.rgw.otp", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "" 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "default.rgw.buckets.index", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "default.rgw.buckets.data", 2024-06-15T07:37:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "", 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "default.rgw.log:notif" 2024-06-15T07:37:19.500 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:19.501 DEBUG:tasks.util.rgw: json result: {'id': '9ba29813-939b-4814-b0a1-c7153b87e285', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-06-15T07:37:19.501 INFO:tasks.rgw:Starting rgw... 2024-06-15T07:37:19.501 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-06-15T07:37:19.501 INFO:tasks.rgw:Using beast as radosgw frontend 2024-06-15T07:37:19.502 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-06-15T07:37:19.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-06-15T07:37:19.505 INFO:tasks.rgw.c1.client.0:Started 2024-06-15T07:37:19.505 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-06-15T07:37:19.505 INFO:tasks.rgw:Using beast as radosgw frontend 2024-06-15T07:37:19.505 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-06-15T07:37:19.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-06-15T07:37:19.507 INFO:tasks.rgw.c1.client.1:Started 2024-06-15T07:37:19.507 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-06-15T07:37:19.507 INFO:tasks.rgw:Using beast as radosgw frontend 2024-06-15T07:37:19.508 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-06-15T07:37:19.508 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-06-15T07:37:19.543 INFO:tasks.rgw.c2.client.0:Started 2024-06-15T07:37:19.544 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-06-15T07:37:19.544 INFO:tasks.rgw:Using beast as radosgw frontend 2024-06-15T07:37:19.544 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-06-15T07:37:19.544 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-06-15T07:37:19.546 INFO:tasks.rgw.c2.client.1:Started 2024-06-15T07:37:19.546 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi012.front.sepia.ceph.com:8000/ 2024-06-15T07:37:19.546 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000/ 2024-06-15T07:37:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-06-15T07:37:19.563 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:19.563 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:19.563 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-06-15T07:37:19.563 INFO:teuthology.orchestra.run.smithi012.stderr:curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-06-15T07:37:20.565 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000/ 2024-06-15T07:37:20.577 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:20.577 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:20.579 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 80937 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:20.580 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2024-06-15T07:37:20.580 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi012.front.sepia.ceph.com:8001/ 2024-06-15T07:37:20.581 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001/ 2024-06-15T07:37:20.636 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:20.636 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:20.638 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 98617 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:20.639 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2024-06-15T07:37:20.639 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi131.front.sepia.ceph.com:8000/ 2024-06-15T07:37:20.639 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000/ 2024-06-15T07:37:20.660 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:20.660 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:20.664 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 77061 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:20.665 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2024-06-15T07:37:20.665 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi131.front.sepia.ceph.com:8001/ 2024-06-15T07:37:20.665 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8001/ 2024-06-15T07:37:20.716 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:20.716 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:20.719 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71001 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:20.720 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2024-06-15T07:37:20.720 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-06-15T07:37:20.780 INFO:tasks.rgw_multisite:creating realm.. 2024-06-15T07:37:20.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.781 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:20.821 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:20.821 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:20.856 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-realm", 2024-06-15T07:37:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "current_period": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T07:37:20.857 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1 2024-06-15T07:37:20.857 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:20.858 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-06-15T07:37:20.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.859 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:20.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:20.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:20.955+0000 7f586d320a40 -1 old format 2024-06-15T07:37:20.969 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:20.969 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "test-zonegroup", 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zonegroup", 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "", 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": true, 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "", 2024-06-15T07:37:20.970 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [], 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [] 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:20.971 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T07:37:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:37:20.972 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:37:20.972 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:20.973 INFO:tasks.rgw_multisite:creating master zone.. 2024-06-15T07:37:20.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:20.973 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key JU2SIBEYF8X99UFF --secret 8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ --endpoints http://smithi012.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:21.056 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:21.057 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:21.075 INFO:teuthology.orchestra.run.smithi012.stderr:NOTICE: set zone's realm_id=c43304c1-0b90-4ac6-a459-45e84b6191f4 2024-06-15T07:37:21.096 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "test-zone1.rgw.control", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "test-zone1.rgw.log", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-06-15T07:37:21.097 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:21.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:21.099 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-06-15T07:37:21.100 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:21.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2024-06-15T07:37:21.796 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'test-zone1.rgw.buckets.data' created 2024-06-15T07:37:21.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2024-06-15T07:37:22.804 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2024-06-15T07:37:22.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:22.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:22.862 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:22.899 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:22.900 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "test-zone1.rgw.meta:root", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "test-zone1.rgw.control", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "test-zone1.rgw.log", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2024-06-15T07:37:22.926 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "test-zone1.rgw.otp", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:22.927 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "compression_type": "random" 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:22.928 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:22.929 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:22.929 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2024-06-15T07:37:22.929 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:22.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:22.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:22.929 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:23.001 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:23.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:23.015+0000 7f8582001a40 0 period (6efc1a3f-742f-4f4f-9712-58131a5a0680 does not have zone 7ac29e95-fe0d-461f-ae1e-cb1f85586a02 configured 2024-06-15T07:37:29.931 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 1, 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": [], 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "period_map": { 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "zonegroups": [ 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "test-zonegroup", 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zonegroup", 2024-06-15T07:37:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "", 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": true, 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [ 2024-06-15T07:37:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": false, 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T07:37:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [ 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD" 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:37:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "short_zone_ids": [ 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 3768332864 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "period_config": { 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:37:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "user_ratelimit": { 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:37:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_ratelimit": { 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "anonymous_ratelimit": { 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:37:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_name": "test-realm", 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:37:29.940 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:29.941 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-06-15T07:37:29.941 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T07:37:29.988 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T07:37:29.987+0000 7f65d5f64640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 27522) UID: 0 2024-06-15T07:37:29.989 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T07:37:29.987+0000 7f65d97aba40 -1 shutting down 2024-06-15T07:37:34.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-06-15T07:37:36.043 INFO:tasks.rgw.c1.client.0:Stopped 2024-06-15T07:37:36.043 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-06-15T07:37:36.043 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-06-15T07:37:36.046 INFO:tasks.rgw.c1.client.0:Started 2024-06-15T07:37:36.046 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000 2024-06-15T07:37:36.058 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:36.058 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:36.059 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-06-15T07:37:36.059 INFO:teuthology.orchestra.run.smithi012.stderr:curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-06-15T07:37:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-06-15T07:37:37.060 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000 2024-06-15T07:37:37.071 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:37.072 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:37.074 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 69934 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:37.075 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2024-06-15T07:37:37.076 INFO:tasks.rgw_multisite:creating admin user.. 2024-06-15T07:37:37.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.076 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key JU2SIBEYF8X99UFF --secret 8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:37.171 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:37.212 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:37.212 INFO:teuthology.orchestra.run.smithi012.stdout: "user_id": "realm-admin", 2024-06-15T07:37:37.212 INFO:teuthology.orchestra.run.smithi012.stdout: "display_name": "Realm Admin", 2024-06-15T07:37:37.212 INFO:teuthology.orchestra.run.smithi012.stdout: "email": "", 2024-06-15T07:37:37.212 INFO:teuthology.orchestra.run.smithi012.stdout: "suspended": 0, 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "max_buckets": 1000, 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "subusers": [], 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "keys": [ 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "user": "realm-admin", 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "swift_keys": [], 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": [], 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "op_mask": "read, write, delete", 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "system": true, 2024-06-15T07:37:37.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "", 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "default_storage_class": "", 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_tags": [], 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:37:37.214 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "temp_url_keys": [], 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "rgw", 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: "mfa_ids": [] 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:37.215 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:37:37.227 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-06-15T07:37:37.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.227 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi012.front.sepia.ceph.com:8000 --access-key JU2SIBEYF8X99UFF --secret 8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:37.274 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:37.274 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:37.319 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-realm", 2024-06-15T07:37:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "current_period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:37:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 2 2024-06-15T07:37:37.320 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:37.321 INFO:tasks.rgw_multisite:creating zone.. 2024-06-15T07:37:37.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:37.322 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key JU2SIBEYF8X99UFF --secret 8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ --endpoints http://smithi131.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:37.362 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:37.362 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:37.380 INFO:teuthology.orchestra.run.smithi131.stderr:NOTICE: set zone's realm_id=c43304c1-0b90-4ac6-a459-45e84b6191f4 2024-06-15T07:37:37.397 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone2", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "test-zone2.rgw.control", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-06-15T07:37:37.398 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "test-zone2.rgw.log", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-06-15T07:37:37.399 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-06-15T07:37:37.400 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-06-15T07:37:37.401 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:37.402 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2024-06-15T07:37:38.142 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'test-zone2.rgw.buckets.data' created 2024-06-15T07:37:38.182 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2024-06-15T07:37:39.153 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2024-06-15T07:37:39.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:39.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:39.210 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:39.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:39.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:39.267 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:39.267 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:37:39.267 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone2", 2024-06-15T07:37:39.267 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "test-zone2.rgw.control", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "test-zone2.rgw.log", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-06-15T07:37:39.268 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2024-06-15T07:37:39.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-06-15T07:37:39.270 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:39.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:39.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:39.271 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:39.348 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:39.349 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:39.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:37:39.363+0000 7f13705ffa40 0 period (6634f146-140c-4941-81ef-ff44e7855a30 does not have zone 75a221df-8373-409f-a0c0-681fe8b4f342 configured 2024-06-15T07:37:39.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-06-15T07:37:46.260 INFO:teuthology.orchestra.run.smithi131.stderr:Sending period to new master zone 7ac29e95-fe0d-461f-ae1e-cb1f85586a02 2024-06-15T07:37:46.306 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 2, 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": [], 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "period_map": { 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "zonegroups": [ 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "test-zonegroup", 2024-06-15T07:37:46.307 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zonegroup", 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": true, 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2024-06-15T07:37:46.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.309 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:37:46.309 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone2", 2024-06-15T07:37:46.309 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T07:37:46.309 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2024-06-15T07:37:46.310 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone1", 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T07:37:46.311 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2024-06-15T07:37:46.312 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.313 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "short_zone_ids": [ 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:37:46.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 698550714 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3768332864 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "period_config": { 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_quota": { 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2024-06-15T07:37:46.315 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "user_quota": { 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "user_ratelimit": { 2024-06-15T07:37:46.316 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_ratelimit": { 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T07:37:46.317 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "anonymous_ratelimit": { 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T07:37:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_name": "test-realm", 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:37:46.330 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:37:46.332 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-06-15T07:37:46.333 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T07:37:46.376 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T07:37:46.375+0000 7f9059ddb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 27402) UID: 0 2024-06-15T07:37:46.376 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T07:37:46.375+0000 7f905d622a40 -1 shutting down 2024-06-15T07:37:51.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-06-15T07:37:52.434 INFO:tasks.rgw.c2.client.0:Stopped 2024-06-15T07:37:52.434 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-06-15T07:37:52.435 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-06-15T07:37:52.438 INFO:tasks.rgw.c2.client.0:Started 2024-06-15T07:37:52.438 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000 2024-06-15T07:37:52.449 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:52.449 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:52.449 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-06-15T07:37:52.449 INFO:teuthology.orchestra.run.smithi131.stderr:curl: (7) Failed to connect to smithi131.front.sepia.ceph.com port 8000 after 0 ms: Connection refused 2024-06-15T07:37:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-06-15T07:37:53.451 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000 2024-06-15T07:37:53.462 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:37:53.463 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:37:53.465 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 79259 0 --:--:-- --:--:-- --:--:-- 104k 2024-06-15T07:37:53.466 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2024-06-15T07:37:53.466 INFO:tasks.rgw_multisite:creating zone.. 2024-06-15T07:37:53.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:53.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'JU2SIBEYF8X99UFF', '--secret', '8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:53.466 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key JU2SIBEYF8X99UFF --secret 8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ --endpoints http://smithi012.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:53.515 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:53.536 INFO:teuthology.orchestra.run.smithi012.stderr:NOTICE: set zone's realm_id=c43304c1-0b90-4ac6-a459-45e84b6191f4 2024-06-15T07:37:53.554 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone3", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "test-zone3.rgw.control", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "test-zone3.rgw.log", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-06-15T07:37:53.555 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-06-15T07:37:53.556 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-06-15T07:37:53.557 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:53.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2024-06-15T07:37:54.833 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'test-zone3.rgw.buckets.data' created 2024-06-15T07:37:54.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2024-06-15T07:37:55.828 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2024-06-15T07:37:55.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:55.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:55.900 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:55.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:55.937 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:55.960 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:37:55.960 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T07:37:55.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone3", 2024-06-15T07:37:55.960 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "test-zone3.rgw.meta:root", 2024-06-15T07:37:55.960 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "test-zone3.rgw.control", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "test-zone3.rgw.log", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "test-zone3.rgw.otp", 2024-06-15T07:37:55.961 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2024-06-15T07:37:55.962 INFO:teuthology.orchestra.run.smithi012.stdout: "compression_type": "random" 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2024-06-15T07:37:55.963 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:37:55.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:55.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:37:55.965 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:37:56.048 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:37:56.048 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:37:56.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:37:56.062+0000 7fc8b0145a40 0 period (6634f146-140c-4941-81ef-ff44e7855a30 does not have zone ec1d75a7-d3e8-402a-8fe3-454f79b9554f configured 2024-06-15T07:37:56.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-06-15T07:38:02.959 INFO:teuthology.orchestra.run.smithi012.stderr:Sending period to new master zone 7ac29e95-fe0d-461f-ae1e-cb1f85586a02 2024-06-15T07:38:02.981 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 3, 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": [], 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "period_map": { 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "zonegroups": [ 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.982 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "test-zonegroup", 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zonegroup", 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "", 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": true, 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [ 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:38:02.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone2", 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T07:38:02.984 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T07:38:02.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T07:38:02.986 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone3", 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8001" 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T07:38:02.987 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:02.988 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [ 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD" 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:38:02.989 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "short_zone_ids": [ 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T07:38:02.990 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 698550714 2024-06-15T07:38:02.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.991 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:38:02.991 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 3768332864 2024-06-15T07:38:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.995 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 2892951031 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "period_config": { 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:38:02.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "user_ratelimit": { 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:38:02.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_ratelimit": { 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "anonymous_ratelimit": { 2024-06-15T07:38:02.998 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_name": "test-realm", 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:38:02.999 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:38:03.000 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-06-15T07:38:03.000 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T07:38:03.046 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T07:38:03.042+0000 7f685e7e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 27525) UID: 0 2024-06-15T07:38:03.046 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T07:38:03.042+0000 7f6862029a40 -1 shutting down 2024-06-15T07:38:08.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-06-15T07:38:09.102 INFO:tasks.rgw.c1.client.1:Stopped 2024-06-15T07:38:09.102 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-06-15T07:38:09.103 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-06-15T07:38:09.106 INFO:tasks.rgw.c1.client.1:Started 2024-06-15T07:38:09.106 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2024-06-15T07:38:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-06-15T07:38:09.118 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:38:09.118 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:38:09.118 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-06-15T07:38:09.119 INFO:teuthology.orchestra.run.smithi012.stderr:curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8001 after 0 ms: Connection refused 2024-06-15T07:38:10.119 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2024-06-15T07:38:10.132 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-06-15T07:38:10.132 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2024-06-15T07:38:10.135 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 67126 0 --:--:-- --:--:-- --:--:-- 71333 2024-06-15T07:38:10.135 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2024-06-15T07:38:10.136 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-06-15T07:38:10.136 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-06-15T07:38:10.148 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi 2024-06-15T07:38:10.180 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-06-15T07:38:10.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:38:10.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T07:38:10.181 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T07:38:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:10.234 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:10.277 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:38:10.277 INFO:teuthology.orchestra.run.smithi012.stdout: "user_id": "rgw-multisite-test-user", 2024-06-15T07:38:10.277 INFO:teuthology.orchestra.run.smithi012.stdout: "display_name": "Multisite Test User", 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "email": "", 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "suspended": 0, 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max_buckets": 1000, 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "subusers": [], 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "keys": [ 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "user": "rgw-multisite-test-user", 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "MAZ73J2OM0E5TW39SGLF", 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "zf14qD3iKpfhB4WX7mLGV2qeMaSWECFxDfdheX3n" 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "swift_keys": [], 2024-06-15T07:38:10.278 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": [ 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "roles", 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "perm": "*" 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "op_mask": "read, write, delete", 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "", 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_storage_class": "", 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_tags": [], 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:38:10.279 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "temp_url_keys": [], 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "rgw", 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout: "mfa_ids": [] 2024-06-15T07:38:10.280 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:38:10.281 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-06-15T07:38:10.292 INFO:rgw_multi.tests:meta checkpoint 2024-06-15T07:38:10.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:10.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:10.293 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:10.325 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:10.326 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:10.362 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:38:10.376 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:38:10.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:38:10.378 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:38:10.378 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:38:10.378 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:38:10.378 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:38:10.378 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:38:10.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:38:10.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:10.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:10.379 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:10.422 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:10.423 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:10.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:10.458+0000 7f1a0cd9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:10.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:10.458+0000 7f1a0cd9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:10.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:10.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:10.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:10.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:10.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:10.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:10.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:10.482 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:10.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:10.482 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:10Z" 2024-06-15T07:38:10.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:10.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:13.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-06-15T07:38:15.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:15.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:15.485 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:15.528 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:15.528 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:15.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:15.558+0000 7fcbe5683a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:15.564 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:15.562+0000 7fcbe5683a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:15.578 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:15.578 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:15.578 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:15.578 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:15.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:15.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:15.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:15.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:15Z" 2024-06-15T07:38:15.583 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:15.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:20.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:20.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:20.586 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:20.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:20.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:20.668 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:20.666+0000 7f7563d91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:20.668 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:20.666+0000 7f7563d91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:20.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:20.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:20.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:20.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:20Z" 2024-06-15T07:38:20.687 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:20.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:25.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:25.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:25.690 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:25.732 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:25.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:25.769 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:25.766+0000 7f5373b49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:25.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:25.766+0000 7f5373b49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:25.784 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:25.784 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:25.784 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:25.784 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:25.784 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:25.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:25.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:25.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:25.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:25Z" 2024-06-15T07:38:25.789 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:25.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:30.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:30.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:30.791 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:30.838 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:30.838 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:30.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:30.870+0000 7fc2c2fcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:30.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:30.874+0000 7fc2c2fcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.889 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:30.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:30.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:30.893 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:30.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:30.894 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:30Z" 2024-06-15T07:38:30.894 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:30.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:35.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:35.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:35.896 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:35.938 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:35.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:35.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:35.970+0000 7f9d264a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:35.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:35.970+0000 7f9d264a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:35.987 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:35.988 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:35.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:35.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:35.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:35Z" 2024-06-15T07:38:35.992 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:35.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:40.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:40.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:40.994 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:41.041 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:41.041 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:41.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:41.082+0000 7f5e4e33fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:41.081 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:41.082+0000 7f5e4e33fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.095 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:41.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:41.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:41.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:41.099 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:41Z" 2024-06-15T07:38:41.100 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:41.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:46.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:46.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:46.102 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:46.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:46.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:46.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:46.181+0000 7f285abf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:46.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:46.181+0000 7f285abf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.196 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:46.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:46.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:46.200 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:46.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:46.201 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:46Z" 2024-06-15T07:38:46.201 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:46.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:51.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:51.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:51.203 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:51.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:51.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:51.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:51.281+0000 7f7909d4ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:51.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:51.281+0000 7f7909d4ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:51.296 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:51.297 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:51.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:51.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:51.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:51Z" 2024-06-15T07:38:51.301 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:51.302 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:38:56.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:56.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:38:56.303 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:38:56.346 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:38:56.346 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:38:56.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:56.385+0000 7f89c695ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:56.385 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:38:56.385+0000 7f89c695ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:38:56.398 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:56.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:56.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:56.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:38:56.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:38:56Z" 2024-06-15T07:38:56.403 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:38:56.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:01.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:01.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:01.406 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:01.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:01.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:01.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:01.485+0000 7f269ac38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:01.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:01.485+0000 7f269ac38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:01.500 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:01.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:01.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:01.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:01Z" 2024-06-15T07:39:01.509 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:01.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:06.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:06.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:06.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:06.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:06.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:06.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:06.589+0000 7f5190d38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:06.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:06.593+0000 7f5190d38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:06.606 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:06.606 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:06.607 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:06.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:06.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:06.610 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:06Z" 2024-06-15T07:39:06.611 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:06.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:11.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:11.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:11.614 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:11.657 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:11.657 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:11.693 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:11.693+0000 7f681aa2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:11.694 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:11.693+0000 7f681aa2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:11.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:11.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:11.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:11.711 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:11Z" 2024-06-15T07:39:11.712 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:11.713 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:16.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:16.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:16.715 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:16.758 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:16.758 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:16.792 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:16.793+0000 7fb728216a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:16.793 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:16.793+0000 7fb728216a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.807 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:16.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:16.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:16.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:16.811 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:16Z" 2024-06-15T07:39:16.812 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:16.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:21.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:21.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:21.814 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:21.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:21.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:21.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:21.897+0000 7fde8db40a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:21.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:21.897+0000 7fde8db40a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:21.910 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:21.910 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:21.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:21.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:21.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:21Z" 2024-06-15T07:39:21.915 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:21.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:26.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:26.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:26.917 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:26.961 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:26.961 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:26.996 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:26.997+0000 7f2446fe7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:26.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:26.997+0000 7f2446fe7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:27Z" 2024-06-15T07:39:27.016 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:27.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:32.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:32.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:32.019 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:32.062 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:32.062 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:32.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:32.097+0000 7f51457e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:32.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:32.097+0000 7f51457e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:32.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:32.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:32.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:32.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:32Z" 2024-06-15T07:39:32.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:32.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:37.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:37.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:37.120 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:37.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:37.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:37.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:37.196+0000 7fca3f3bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:37.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:37.200+0000 7fca3f3bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:37.213 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:37.213 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:37.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:37.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:37.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:37.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:37Z" 2024-06-15T07:39:37.218 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:37.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:42.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:42.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:42.220 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:42.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:42.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:42.299 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:42.296+0000 7faccf802a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:42.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:42.300+0000 7faccf802a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:42.316 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:42.316 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:42.316 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:42.316 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:42.316 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:42.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:42.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:42.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:42.320 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:42Z" 2024-06-15T07:39:42.321 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:42.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:47.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:47.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:47.323 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:47.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:47.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:47.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:47.404+0000 7f56cab21a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:47.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:47.404+0000 7f56cab21a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:47.419 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:47.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:47.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:47.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:47.423 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:47.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:47.424 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:47Z" 2024-06-15T07:39:47.424 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:47.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:52.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:52.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:52.426 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:52.469 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:52.469 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:52.505 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:52.504+0000 7f097c757a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:52.506 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:52.504+0000 7f097c757a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:52.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:52.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:52.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:52Z" 2024-06-15T07:39:52.525 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:52.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:39:57.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:57.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:39:57.527 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:39:57.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:39:57.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:39:57.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:57.604+0000 7f6de487ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:57.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:39:57.604+0000 7f6de487ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:39:57.620 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:39:57.620 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:57.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:57.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:57.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:39:57.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:39:57Z" 2024-06-15T07:39:57.625 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:39:57.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:40:02.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:02.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:02.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:02.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:02.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:02.706 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:02.704+0000 7f97ac46aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:02.707 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:02.708+0000 7f97ac46aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:02.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:02.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:02.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:02.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:02.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:02.726 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:02Z" 2024-06-15T07:40:02.726 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:02.727 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1718437090.276409_28.1 target= 2024-06-15T07:40:07.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.728 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:07.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:07.771 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:07.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:07.804+0000 7fc2383d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:07.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:07.804+0000 7fc2383d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:07.818 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:07.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:07.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:07Z" 2024-06-15T07:40:07.823 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:07.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:07.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:07.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.824 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:07.870 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:07.870 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:07.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:07.905+0000 7f8b43d20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:07.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:07.909+0000 7f8b43d20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:07.922 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:07.922 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:07.922 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:07.922 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:07.923 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:07.924 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:07.925 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:07.927 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:07.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:07.927 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:07Z" 2024-06-15T07:40:07.927 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:07.928 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:07.928 INFO:rgw_multi.tests:meta checkpoint 2024-06-15T07:40:07.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:07.928 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:08.011 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:08.054 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:08.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.070 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.070 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.070 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:08.070 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.070 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:08.071 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:08.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:08.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:08.071 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:08.115 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:08.115 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:08.152 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:08.152+0000 7f60748caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:08.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:08.152+0000 7f60748caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:08.166 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:08.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.170 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:08.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:08.171 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:08.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:08.171 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:08Z" 2024-06-15T07:40:08.171 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:08.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:08.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:08.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:08.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:08.172 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:08.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:08.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:08.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:08.253+0000 7f9017d53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:08.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:08.253+0000 7f9017d53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:08.271 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:08.272 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:08.273 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:08Z" 2024-06-15T07:40:08.274 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:08.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:08.275 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-06-15T07:40:08.307 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-1 2024-06-15T07:40:10.492 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-2 2024-06-15T07:40:13.229 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-3 2024-06-15T07:40:13.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:13.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:13.291 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:13.332 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:13.333 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:13.379 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:13.394 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:13.394 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:13.394 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:13.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:13.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:13.396 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:13.396 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:13.396 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:13.396 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:13.397 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437213.265173_70.1", 2024-06-15T07:40:13.397 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:13.265173Z" 2024-06-15T07:40:13.397 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:13.397 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:13.398 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:13.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:13.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:13.399 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:13.442 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:13.442 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:13.477 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:13.476+0000 7f45bec67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:13.478 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:13.476+0000 7f45bec67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:13.492 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:13.492 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:13.492 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:13.492 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:13.492 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:13.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:13.495 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.496 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437210.443937_59.1", 2024-06-15T07:40:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:10.443937Z", 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:13Z" 2024-06-15T07:40:13.499 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:13.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1718437213.265173_70.1 target=1_1718437210.443937_59.1 2024-06-15T07:40:18.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.503 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:18.546 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:18.546 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:18.583 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:18.584+0000 7fdb58660a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:18.584 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:18.584+0000 7fdb58660a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:18.598 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:18.598 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:18.598 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:18.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:18.600 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:18.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437213.265173_70.1", 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:13.265173Z", 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:18Z" 2024-06-15T07:40:18.603 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:18.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:18.604 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:18.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.605 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:18.649 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:18.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:18.689+0000 7fbfc884ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:18.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:18.689+0000 7fbfc884ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.705 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:18.706 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:18.707 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437213.265173_70.1", 2024-06-15T07:40:18.708 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:13.265173Z", 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:18Z" 2024-06-15T07:40:18.709 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:18.710 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:18.735 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-06-15T07:40:18.735 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-4 2024-06-15T07:40:18.750 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-5 2024-06-15T07:40:18.787 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-6 2024-06-15T07:40:18.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.826 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:18.866 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:18.867 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:18.915 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:18.937 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:18.937 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.937 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:18.937 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:18.807932Z" 2024-06-15T07:40:18.938 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:18.939 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:18.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:18.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:18.940 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:18.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:18.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:19.020+0000 7f52da58ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:19.020+0000 7f52da58ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:19.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:18.807932Z", 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.039 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:19Z" 2024-06-15T07:40:19.040 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:19.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:19.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:19.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.041 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:19.085 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:19.125+0000 7fab08268a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:19.125+0000 7fab08268a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.139 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:19.140 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.141 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.142 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.143 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:18.807932Z", 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:19.144 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.145 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:19Z" 2024-06-15T07:40:19.145 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:19.146 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:19.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.441 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:19.484 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.524 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.541 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:18.807932Z" 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.542 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:19.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:19.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.543 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:19.586 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:19.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:19.624+0000 7f0daf2fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.626 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:19.624+0000 7f0daf2fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:19.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:19.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:18.807932Z", 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:19.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:19.645 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:19.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:19.645 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:19Z" 2024-06-15T07:40:19.645 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:19.646 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:19.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:19.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.647 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:19.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:19.692 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:19.729+0000 7fd8e2620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:19.729+0000 7fd8e2620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:19.746 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.747 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.748 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.749 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:19.750 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437218.807932_82.1", 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:18.807932Z", 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:19.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:19Z" 2024-06-15T07:40:19.752 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:19.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:19.777 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-06-15T07:40:19.777 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-7 2024-06-15T07:40:19.797 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-8 2024-06-15T07:40:19.834 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-9 2024-06-15T07:40:19.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.878 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:19.923 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:19.977 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:19.990 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:19.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437219.859211_94.1", 2024-06-15T07:40:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:19.859211Z" 2024-06-15T07:40:19.992 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:19.992 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:19.993 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:19.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:19.993 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:20.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:20.036 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:20.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:20.068+0000 7fc8e3111a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:20.072+0000 7fc8e3111a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:20.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437219.859211_94.1", 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:19.859211Z", 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:20Z" 2024-06-15T07:40:20.090 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:20.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:20.091 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:20.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.091 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:20.135 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:20.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:20.169+0000 7f78781c6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:20.169+0000 7f78781c6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:20.185 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.186 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:20.187 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:20.188 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437219.859211_94.1", 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:19.859211Z", 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.189 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:20Z" 2024-06-15T07:40:20.190 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:20.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:20.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.532 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:20.576 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:20.613 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:20.628 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:20.629 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:20.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:20.630 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:20.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:20.672 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:20.707 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:20.708+0000 7f7ac1359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:20.708+0000 7f7ac1359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:20.722 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:20.722 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:20.722 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:20.722 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.231572_95.1", 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.231572Z", 2024-06-15T07:40:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:20Z" 2024-06-15T07:40:20.727 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:20.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1718437220.458611_106.1 target=1_1718437220.231572_95.1 2024-06-15T07:40:25.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:25.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:25.730 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:25.772 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:25.773 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:25.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:25.808+0000 7f8d8e060a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:25.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:25.812+0000 7f8d8e060a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:25.825 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:25.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:25.827 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:25.828 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:25.829 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:25Z" 2024-06-15T07:40:25.830 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:25.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:25.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:25.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:25.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:25.831 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:25.875 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:25.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:25.909+0000 7f1577598a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:25.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:25.913+0000 7f1577598a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:25.926 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:25.927 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:25.928 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.929 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.930 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:25.931 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:25.931 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:25.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:25.931 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:25Z" 2024-06-15T07:40:25.931 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:25.932 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:25.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-06-15T07:40:25.943 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-10 2024-06-15T07:40:25.959 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-11 2024-06-15T07:40:26.001 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-12 2024-06-15T07:40:26.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:26.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:26.449 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:26.491 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:26.532 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:40:26.548 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:40:26.548 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437226.024838_44.1", 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:26.024838Z" 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:26.549 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:26.550 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:40:26.550 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:26.550 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:40:26.551 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:40:26.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:26.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:26.551 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:26.595 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:26.595 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:26.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:26.632+0000 7f5edfdbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:26.633 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:26.632+0000 7f5edfdbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:26.647 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437225.952954_33.1", 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:25.952954Z", 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:40:26.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:26Z" 2024-06-15T07:40:26.652 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:26.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718437226.024838_44.1 target=1_1718437225.952954_33.1 2024-06-15T07:40:31.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:31.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:31.655 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:31.698 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:31.699 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:31.736 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:31.736+0000 7f4423fb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:31.737 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:31.736+0000 7f4423fb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:31.751 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:40:31.751 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:40:31.751 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:40:31.751 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:40:31.751 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:40:31.752 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:31.753 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437226.024838_44.1", 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:31.754 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:26.024838Z", 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:40:31.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:40:31Z" 2024-06-15T07:40:31.756 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:40:31.757 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:40:31.757 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:40:31.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:31.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:40:31.758 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:40:31.803 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:31.803 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:31.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:31.841+0000 7fcc58c0ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:31.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:31.841+0000 7fcc58c0ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:31.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:31.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:31.858 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437226.024838_44.1", 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:26.024838Z", 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:40:31.859 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:40:31.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:40:31.861 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:40:31Z" 2024-06-15T07:40:31.861 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:40:31.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:40:31.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:31.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:31.862 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:40:31.911 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:31.911 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:31.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:31.952+0000 7fb1c2829a40 1 waiting to reach incremental sync.. 2024-06-15T07:40:36.953 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:36.952+0000 7fb1c2829a40 1 waiting to reach incremental sync.. 2024-06-15T07:40:41.953 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:41.952+0000 7fb1c2829a40 1 waiting to reach incremental sync.. 2024-06-15T07:40:46.957 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:46.956+0000 7fb1c2829a40 1 bucket sync caught up with source: 2024-06-15T07:40:46.957 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.8.6, , 00000000001.3.6, , , , 00000000001.5.6, 00000000001.3.6, , ] 2024-06-15T07:40:46.957 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000001.8.6, , 00000000001.3.6, , , , 00000000001.5.6, 00000000001.3.6, , ] 2024-06-15T07:40:46.957 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:40:46.956+0000 7fb1c2829a40 0 bucket checkpoint complete 2024-06-15T07:40:46.986 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-10 zones={test-zone2, test-zone1} 2024-06-15T07:40:46.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:40:47.000 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:40:47.000 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:40:47.000 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:40:47.000 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:40:47.000 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:40:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:40:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:40:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:40:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:40:47.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:47.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:47.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:47.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:47.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:47.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:47.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:47.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:47.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:47.699 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-10 zones={test-zone2, test-zone1} 2024-06-15T07:40:47.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:47.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:47.700 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:40:47.751 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:47.751 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:47.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:47.793+0000 7f08fd082a40 1 bucket sync caught up with source: 2024-06-15T07:40:47.796 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.8.6, , 00000000001.3.6, , , , 00000000001.5.6, 00000000001.3.6, , ] 2024-06-15T07:40:47.796 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.8.6, , 00000000001.3.6, , , , 00000000001.5.6, 00000000001.3.6, , ] 2024-06-15T07:40:47.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:47.793+0000 7f08fd082a40 0 bucket checkpoint complete 2024-06-15T07:40:47.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-10 zones={test-zone3, test-zone1} 2024-06-15T07:40:47.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:40:47.817 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:40:47.818 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:40:47.818 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:40:47.818 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:40:47.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:40:47.820 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:40:47.820 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:40:47.821 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:40:47.821 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:40:47.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:40:47.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:48.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:48.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:40:48.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:48.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:48.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:40:48.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:48.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:48.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:40:48.415 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-10 zones={test-zone3, test-zone1} 2024-06-15T07:40:48.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:48.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:40:48.416 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:40:48.467 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:40:48.468 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:40:48.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:48.509+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:40:53.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:53.513+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:40:58.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:40:58.513+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:03.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:03.513+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:08.513+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:13.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:13.513+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:18.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:18.517+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:23.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:23.517+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:28.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:28.517+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:33.520 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:33.517+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:38.521 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:38.521+0000 7fea6c354a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:43.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:43.525+0000 7fea6c354a40 1 bucket sync caught up with source: 2024-06-15T07:41:43.525 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.8.6, , 00000000001.8.6, , , , 00000000001.7.6, 00000000001.9.6, , ] 2024-06-15T07:41:43.525 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.8.6, , 00000000001.8.6, , , , 00000000001.7.6, 00000000001.9.6, , ] 2024-06-15T07:41:43.525 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:43.525+0000 7fea6c354a40 0 bucket checkpoint complete 2024-06-15T07:41:43.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-11 zones={test-zone1, test-zone2} 2024-06-15T07:41:43.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:41:43.568 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:43.568 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:43.568 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:43.568 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:43.569 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:41:43.573 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:43.573 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:43.573 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:43.573 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:43.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:43.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:43.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:43.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:43.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:43.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:43.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:43.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:43.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:44.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-11 zones={test-zone1, test-zone2} 2024-06-15T07:41:44.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:44.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:44.248 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:41:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:44.302 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:44.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:44.353+0000 7f1e316faa40 1 bucket sync caught up with source: 2024-06-15T07:41:44.355 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.8.6, , 00000000001.8.6, , , , 00000000001.7.6, 00000000001.9.6, , ] 2024-06-15T07:41:44.355 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.8.6, , 00000000001.8.6, , , , 00000000001.7.6, 00000000001.9.6, , ] 2024-06-15T07:41:44.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:44.353+0000 7f1e316faa40 0 bucket checkpoint complete 2024-06-15T07:41:44.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-11 zones={test-zone3, test-zone2} 2024-06-15T07:41:44.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:41:44.377 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:44.377 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:44.377 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:44.377 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:44.377 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:41:44.381 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:44.381 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:44.381 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:44.381 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:44.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:44.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:44.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:44.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:44.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:44.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:44.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:44.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:44.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:44.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:44.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-11 zones={test-zone3, test-zone2} 2024-06-15T07:41:44.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:44.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:44.952 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:41:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:45.004 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:45.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:45.053+0000 7f0a2d611a40 1 bucket sync caught up with source: 2024-06-15T07:41:45.052 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.49.6, , 00000000001.6.6, , , , 00000000001.10.6, 00000000001.17.6, , ] 2024-06-15T07:41:45.053 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.49.6, , 00000000001.6.6, , , , 00000000001.10.6, 00000000001.17.6, , ] 2024-06-15T07:41:45.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:45.053+0000 7f0a2d611a40 0 bucket checkpoint complete 2024-06-15T07:41:45.070 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-12 zones={test-zone1, test-zone3} 2024-06-15T07:41:45.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:41:45.077 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:45.077 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:45.078 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:45.078 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:45.078 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:41:45.081 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:45.081 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:45.081 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:45.081 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:45.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:45.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:45.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:45.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:45.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:45.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:45.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:45.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:45.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:45.677 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-12 zones={test-zone1, test-zone3} 2024-06-15T07:41:45.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:45.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:45.677 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:41:45.729 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:45.729 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:45.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:45.771+0000 7f410255da40 1 bucket sync caught up with source: 2024-06-15T07:41:45.774 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.49.6, , 00000000001.6.6, , , , 00000000001.10.6, 00000000001.17.6, , ] 2024-06-15T07:41:45.774 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000001.49.6, , 00000000001.6.6, , , , 00000000001.10.6, 00000000001.17.6, , ] 2024-06-15T07:41:45.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:45.771+0000 7f410255da40 0 bucket checkpoint complete 2024-06-15T07:41:45.789 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-12 zones={test-zone2, test-zone3} 2024-06-15T07:41:45.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:41:45.798 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:45.798 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:45.798 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:45.798 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:45.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:41:45.803 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-06-15T07:41:45.803 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-06-15T07:41:45.803 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-06-15T07:41:45.803 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:41:45.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-06-15T07:41:45.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:45.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:46.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-06-15T07:41:46.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:46.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:46.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-06-15T07:41:46.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:46.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:46.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:41:46.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-12 zones={test-zone2, test-zone3} 2024-06-15T07:41:46.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-06-15T07:41:46.344 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-13 2024-06-15T07:41:46.375 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-14 2024-06-15T07:41:46.430 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-15 2024-06-15T07:41:46.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:46.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:46.568 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:41:46.610 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:46.610 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:46.648 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:41:46.663 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437306.464994_60.1", 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:41:46.464994Z" 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:46.664 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:41:46.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:41:46.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:46.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:46.666 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:41:46.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:46.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:46.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:46.747+0000 7ff58211fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:46.748 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:46.747+0000 7ff58211fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:46.762 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:46.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:41:46.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437306.357355_49.1", 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:46.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:41:46.357355Z", 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:41:46.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:41:46Z" 2024-06-15T07:41:46.767 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:41:46.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718437306.464994_60.1 target=1_1718437306.357355_49.1 2024-06-15T07:41:51.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:51.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:51.770 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:41:51.814 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:51.814 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:51.849 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:51.847+0000 7f939561da40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:51.850 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:51.851+0000 7f939561da40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:51.864 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:51.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:51.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437306.464994_60.1", 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:51.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:41:46.464994Z", 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:41:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:41:51Z" 2024-06-15T07:41:51.869 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:41:51.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:41:51.870 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:41:51.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:51.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:41:51.870 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:41:51.916 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:51.917 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:51.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:51.952+0000 7fdd78198a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:51.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:41:51.952+0000 7fdd78198a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:41:51.971 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:41:51.972 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437306.464994_60.1", 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:41:51.973 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:41:46.464994Z", 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:41:51.974 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:41:51Z" 2024-06-15T07:41:51.975 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:41:51.976 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:41:51.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:51.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:41:51.977 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:41:52.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:41:52.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:41:52.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:52.067+0000 7fa09d8a5a40 1 waiting to reach incremental sync.. 2024-06-15T07:41:57.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:41:57.071+0000 7fa09d8a5a40 1 waiting to reach incremental sync.. 2024-06-15T07:42:02.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:02.071+0000 7fa09d8a5a40 1 waiting to reach incremental sync.. 2024-06-15T07:42:07.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:07.075+0000 7fa09d8a5a40 1 bucket sync caught up with source: 2024-06-15T07:42:07.075 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:07.075 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:07.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:07.075+0000 7fa09d8a5a40 0 bucket checkpoint complete 2024-06-15T07:42:07.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-13 zones={test-zone2, test-zone1} 2024-06-15T07:42:07.111 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:07.162 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.163 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:07.168 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-13 zones={test-zone2, test-zone1} 2024-06-15T07:42:07.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.311 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:07.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.404+0000 7fcd864b5a40 1 bucket sync caught up with source: 2024-06-15T07:42:07.407 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:07.407 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:07.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.404+0000 7fcd864b5a40 0 bucket checkpoint complete 2024-06-15T07:42:07.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-13 zones={test-zone3, test-zone1} 2024-06-15T07:42:07.427 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:07.430 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.430 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:07.433 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.558 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-13 zones={test-zone3, test-zone1} 2024-06-15T07:42:07.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.559 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:07.609 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:07.610 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.652+0000 7f52ad844a40 1 bucket sync caught up with source: 2024-06-15T07:42:07.656 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:07.656 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:07.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.652+0000 7f52ad844a40 0 bucket checkpoint complete 2024-06-15T07:42:07.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-14 zones={test-zone1, test-zone2} 2024-06-15T07:42:07.677 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:07.680 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.680 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:07.684 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:07.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-14 zones={test-zone1, test-zone2} 2024-06-15T07:42:07.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:07.822 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:07.874 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:07.874 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:07.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.920+0000 7f878a552a40 1 bucket sync caught up with source: 2024-06-15T07:42:07.921 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:07.921 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:07.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:07.920+0000 7f878a552a40 0 bucket checkpoint complete 2024-06-15T07:42:07.936 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-14 zones={test-zone3, test-zone2} 2024-06-15T07:42:07.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:07.945 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.945 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:07.990 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:07.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-14 zones={test-zone3, test-zone2} 2024-06-15T07:42:08.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.131 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:08.183 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:08.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:08.224+0000 7fe878275a40 1 bucket sync caught up with source: 2024-06-15T07:42:08.225 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:42:08.225 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:42:08.225 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:08.224+0000 7fe878275a40 0 bucket checkpoint complete 2024-06-15T07:42:08.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-15 zones={test-zone1, test-zone3} 2024-06-15T07:42:08.245 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:08.248 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:08.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:08.250 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:08.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-15 zones={test-zone1, test-zone3} 2024-06-15T07:42:08.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.423 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:08.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:08.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:08.519 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:08.519+0000 7f0e2a214a40 1 bucket sync caught up with source: 2024-06-15T07:42:08.519 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:42:08.519 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:42:08.519 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:08.519+0000 7f0e2a214a40 0 bucket checkpoint complete 2024-06-15T07:42:08.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-15 zones={test-zone2, test-zone3} 2024-06-15T07:42:08.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:08.545 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:08.545 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:08.550 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-06-15T07:42:08.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-06-15T07:42:08.735 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-15 zones={test-zone2, test-zone3} 2024-06-15T07:42:08.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:08.751 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:08.802 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:08.802 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:08.848 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:08.847+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:08.848 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:08.848 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:13.851 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:13.851+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:13.851 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:13.851 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:18.853 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:18.851+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:18.853 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:18.853 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:23.855 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:23.855+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:23.855 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:23.855 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:28.857 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:28.855+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:28.858 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:28.858 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:33.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:33.859+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:33.860 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:33.860 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:38.861 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:38.859+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:38.861 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:38.862 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:43.863 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:43.863+0000 7fd8e33f1a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:43.863 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000001.9.6, , , , , , , , , ] 2024-06-15T07:42:43.864 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:48.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:48.863+0000 7fd8e33f1a40 1 bucket sync caught up with source: 2024-06-15T07:42:48.866 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:48.866 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:48.866 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:42:48.863+0000 7fd8e33f1a40 0 bucket checkpoint complete 2024-06-15T07:42:48.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-13 zones={test-zone2, test-zone1} 2024-06-15T07:42:48.903 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:48.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:48.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-13 zones={test-zone2, test-zone1} 2024-06-15T07:42:48.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:48.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:48.923 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:48.974 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:48.974 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:49.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:49.016+0000 7f0b914b6a40 1 bucket sync caught up with source: 2024-06-15T07:42:49.018 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:49.018 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.11.6, , , , , , , , , ] 2024-06-15T07:42:49.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:49.016+0000 7f0b914b6a40 0 bucket checkpoint complete 2024-06-15T07:42:49.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-13 zones={test-zone3, test-zone1} 2024-06-15T07:42:49.039 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:42:49.042 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:42:49.093 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-13 zones={test-zone3, test-zone1} 2024-06-15T07:42:49.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:49.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:42:49.112 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:42:49.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:42:49.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:42:49.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:49.204+0000 7fc303564a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:49.207 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:49.207 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:42:54.210 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:54.208+0000 7fc303564a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:54.210 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:54.210 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:42:59.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:42:59.212+0000 7fc303564a40 1 waiting for incremental sync to catch up: 2024-06-15T07:42:59.213 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.21.6, , , , , , , , , ] 2024-06-15T07:42:59.213 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:43:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:04.212+0000 7fc303564a40 1 bucket sync caught up with source: 2024-06-15T07:43:04.215 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:43:04.215 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:43:04.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:04.212+0000 7fc303564a40 0 bucket checkpoint complete 2024-06-15T07:43:04.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-14 zones={test-zone1, test-zone2} 2024-06-15T07:43:04.253 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:43:04.258 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:43:04.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-14 zones={test-zone1, test-zone2} 2024-06-15T07:43:04.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:04.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:04.312 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:43:04.365 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:04.366 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:04.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:04.412+0000 7fcf0dfd4a40 1 bucket sync caught up with source: 2024-06-15T07:43:04.414 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:43:04.414 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.25.6, , , , , , , , , ] 2024-06-15T07:43:04.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:04.412+0000 7fcf0dfd4a40 0 bucket checkpoint complete 2024-06-15T07:43:04.431 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-14 zones={test-zone3, test-zone2} 2024-06-15T07:43:04.436 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:43:04.442 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:43:04.494 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-14 zones={test-zone3, test-zone2} 2024-06-15T07:43:04.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:04.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:04.508 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:43:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:04.552 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:04.596 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:04.596+0000 7f89720eda40 1 waiting for incremental sync to catch up: 2024-06-15T07:43:04.596 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:43:04.597 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:09.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:09.596+0000 7f89720eda40 1 waiting for incremental sync to catch up: 2024-06-15T07:43:09.599 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:43:09.599 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:14.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:14.600+0000 7f89720eda40 1 waiting for incremental sync to catch up: 2024-06-15T07:43:14.601 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:43:14.601 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:19.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:19.600+0000 7f89720eda40 1 waiting for incremental sync to catch up: 2024-06-15T07:43:19.604 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000001.13.6, , , , , , , , , ] 2024-06-15T07:43:19.604 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:24.604+0000 7f89720eda40 1 bucket sync caught up with source: 2024-06-15T07:43:24.606 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:24.606 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:24.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:24.604+0000 7f89720eda40 0 bucket checkpoint complete 2024-06-15T07:43:24.633 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-15 zones={test-zone1, test-zone3} 2024-06-15T07:43:24.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:43:24.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:43:24.694 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-15 zones={test-zone1, test-zone3} 2024-06-15T07:43:24.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:24.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:24.695 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:43:24.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:24.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:24.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:24.786+0000 7f03b715ea40 1 bucket sync caught up with source: 2024-06-15T07:43:24.789 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:24.789 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000002.17.6, , , , , , , , , ] 2024-06-15T07:43:24.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:24.786+0000 7f03b715ea40 0 bucket checkpoint complete 2024-06-15T07:43:24.804 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-15 zones={test-zone2, test-zone3} 2024-06-15T07:43:24.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:43:24.813 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:43:24.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-15 zones={test-zone2, test-zone3} 2024-06-15T07:43:24.867 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-06-15T07:43:24.867 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-16 2024-06-15T07:43:24.909 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-17 2024-06-15T07:43:24.972 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-18 2024-06-15T07:43:29.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.539 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:43:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:29.611 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:29.701 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:43:29.716 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437405.007338_74.1", 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:43:25.007338Z" 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:43:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:43:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.718 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:43:29.719 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:43:29.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.719 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:43:29.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:29.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:29.798 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:29.798+0000 7fe60129ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:43:29.799 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:29.798+0000 7fe60129ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:43:29.813 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:43:29.813 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:43:29.813 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:43:29.813 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:43:29.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:43:29.815 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:43:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437405.007338_74.1", 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:43:25.007338Z", 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:43:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:43:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:43:29Z" 2024-06-15T07:43:29.819 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:43:29.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:43:29.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:43:29.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:43:29.821 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:43:29.863 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:29.864 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:29.901 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:29.900+0000 7f20adce0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:43:29.902 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:43:29.900+0000 7f20adce0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:43:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437405.007338_74.1", 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:43:25.007338Z", 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:43:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:43:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:43:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:43:29Z" 2024-06-15T07:43:29.921 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:43:29.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:43:29.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:29.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:43:29.922 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:43:29.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:43:29.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:43:30.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:30.018+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:43:35.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:35.018+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:43:40.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:40.018+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:43:45.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:45.018+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:43:50.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:50.022+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:43:55.023 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:43:55.022+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:00.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:00.022+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:05.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:05.026+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:10.026 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:10.026+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:15.027 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:15.026+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:20.028 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:20.030+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:25.029 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:25.030+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:30.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:30.034+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:35.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:35.034+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:40.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:40.033+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:45.032 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:45.033+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:50.034 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:50.037+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:44:55.034 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:44:55.037+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:00.036 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:00.037+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:05.037 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:05.041+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:10.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:10.041+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:15.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:15.041+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:20.040 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:20.041+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:25.041 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:25.045+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:30.042 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:30.045+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:35.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:35.045+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:40.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:40.045+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:45.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:45.049+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:50.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:50.049+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:45:55.047 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:45:55.049+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:00.048 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:00.049+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:05.048 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:05.053+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:10.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:10.053+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:15.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:15.053+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:20.052 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:20.056+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:25.053 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:25.056+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:30.054 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:30.056+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:35.055 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:35.056+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:40.056 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:40.060+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:45.058 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:45.060+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:50.059 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:50.060+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:46:55.060 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:46:55.064+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:00.061 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:00.064+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:05.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:05.064+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:10.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:10.064+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:15.065 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:15.068+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:20.066 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:20.068+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:25.067 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:25.067+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:30.068 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:30.071+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:35.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:35.071+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:40.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:40.071+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:45.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:45.071+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:50.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:50.075+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:47:55.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:47:55.075+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:00.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:00.075+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:05.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:05.075+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:10.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:10.079+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:15.077 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:15.079+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:20.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:20.083+0000 7f7b3ae98a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:25.079 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:25.083+0000 7f7b3ae98a40 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-06-15T07:48:25.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:25.083+0000 7f7b3ae98a40 0 bucket sync checkpoint failed: (110) Connection timed out 2024-06-15T07:48:25.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:25.083+0000 7f7b3ae98a40 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-06-15T07:48:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-06-15T07:48:25.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-06-15T07:48:25.115 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-19 2024-06-15T07:48:25.150 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-20 2024-06-15T07:48:25.205 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-21 2024-06-15T07:48:25.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:25.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:25.305 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:48:25.348 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:25.348 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:25.393 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:48:25.410 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:48:25.410 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437705.299975_187.1", 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:48:25.299975Z" 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:25.411 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:48:25.412 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:48:25.412 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:25.412 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:48:25.413 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:48:25.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:25.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:25.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:48:25.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:25.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:25.511 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:25.511+0000 7ff009e88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:25.512 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:25.515+0000 7ff009e88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:25.537 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:48:25.537 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:48:25.537 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:48:25.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:25.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437705.169219_174.1", 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:25.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:48:25.169219Z", 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:25.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:48:25Z" 2024-06-15T07:48:25.542 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:48:25.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718437705.299975_187.1 target=1_1718437705.169219_174.1 2024-06-15T07:48:30.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:30.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:30.545 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:48:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:30.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:30.623 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:30.626+0000 7f3e11d58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:30.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:30.626+0000 7f3e11d58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:30.638 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:48:30.638 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:48:30.638 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:48:30.638 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:48:30.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:48:30.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:30.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437705.299975_187.1", 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:48:25.299975Z", 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:48:30.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.643 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:48:30.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:48:30.644 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:48:30.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:48:30.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:48:30Z" 2024-06-15T07:48:30.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:48:30.645 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:48:30.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:48:30.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:30.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:48:30.645 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:48:30.690 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:30.690 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:30.727 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:30.725+0000 7fe7038cda40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:30.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:30.729+0000 7fe7038cda40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437705.299975_187.1", 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:48:25.299975Z", 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:48:30Z" 2024-06-15T07:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:48:30.748 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:48:30.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:30.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:30.763 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:48:30.814 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:30.814 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:30.857 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:30.858+0000 7efe7cdb6a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:35.858 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:35.858+0000 7efe7cdb6a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:40.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:40.858+0000 7efe7cdb6a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:45.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:45.862+0000 7efe7cdb6a40 1 bucket sync caught up with source: 2024-06-15T07:48:45.862 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-06-15T07:48:45.862 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-06-15T07:48:45.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:48:45.862+0000 7efe7cdb6a40 0 bucket checkpoint complete 2024-06-15T07:48:45.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:45.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:45.886 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:48:45.939 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:45.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:45.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:45.981+0000 7ff5f7062a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:50.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:50.981+0000 7ff5f7062a40 1 bucket sync caught up with source: 2024-06-15T07:48:50.984 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.30.11, , , , , ] 2024-06-15T07:48:50.984 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000002.30.11, , , , , ] 2024-06-15T07:48:50.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:50.981+0000 7ff5f7062a40 0 bucket checkpoint complete 2024-06-15T07:48:51.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:51.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:48:51.034 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:48:51.085 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:48:51.085 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:48:51.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:51.129+0000 7f2fd7f39a40 1 waiting to reach incremental sync.. 2024-06-15T07:48:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:48:56.129+0000 7f2fd7f39a40 1 waiting to reach incremental sync.. 2024-06-15T07:49:01.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:01.129+0000 7f2fd7f39a40 1 waiting to reach incremental sync.. 2024-06-15T07:49:06.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:06.133+0000 7f2fd7f39a40 1 bucket sync caught up with source: 2024-06-15T07:49:06.134 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.61.11, , , , , ] 2024-06-15T07:49:06.135 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000002.61.11, , , , , ] 2024-06-15T07:49:06.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:06.133+0000 7f2fd7f39a40 0 bucket checkpoint complete 2024-06-15T07:49:06.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:06.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:06.168 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:49:06.260 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:49:06.261 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:49:06.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:06.301+0000 7f1c9650ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:06.304 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.61.11, , , , , ] 2024-06-15T07:49:06.304 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000003.63.10, , , , , ] 2024-06-15T07:49:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:11.305+0000 7f1c9650ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:11.307 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.61.11, , , , , ] 2024-06-15T07:49:11.307 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000003.63.10, , , , , ] 2024-06-15T07:49:16.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:16.305+0000 7f1c9650ba40 1 bucket sync caught up with source: 2024-06-15T07:49:16.308 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.65.10, , , , , ] 2024-06-15T07:49:16.308 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000003.63.10, , , , , ] 2024-06-15T07:49:16.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:16.305+0000 7f1c9650ba40 0 bucket checkpoint complete 2024-06-15T07:49:16.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:16.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:16.366 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:49:16.416 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:49:16.416 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:49:16.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:16.457+0000 7f4e51e45a40 1 waiting to reach incremental sync.. 2024-06-15T07:49:21.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:21.461+0000 7f4e51e45a40 1 waiting to reach incremental sync.. 2024-06-15T07:49:26.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:26.461+0000 7f4e51e45a40 1 bucket sync caught up with source: 2024-06-15T07:49:26.463 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.44.11, , , , , ] 2024-06-15T07:49:26.463 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000002.44.11, , , , , ] 2024-06-15T07:49:26.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:49:26.461+0000 7f4e51e45a40 0 bucket checkpoint complete 2024-06-15T07:49:26.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:26.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:26.489 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:49:26.539 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:49:26.539 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:49:26.585 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:26.589+0000 7efdbe6dda40 1 bucket sync caught up with source: 2024-06-15T07:49:26.585 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000002.44.11, , , , , ] 2024-06-15T07:49:26.585 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000002.44.11, , , , , ] 2024-06-15T07:49:26.585 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:26.589+0000 7efdbe6dda40 0 bucket checkpoint complete 2024-06-15T07:49:27.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:27.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:49:27.859 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:49:27.909 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:49:27.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:49:27.955 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:27.957+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:27.955 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , , , 00000000003.32.10, , , , , ] 2024-06-15T07:49:27.955 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:32.957 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:32.961+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:32.957 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , , , 00000000003.32.10, , , , , ] 2024-06-15T07:49:32.957 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:37.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:37.965+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:37.960 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , , , 00000000003.32.10, , , , , ] 2024-06-15T07:49:37.960 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:42.961 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:42.965+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:42.962 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , , , 00000000003.32.10, , , , , ] 2024-06-15T07:49:42.962 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:47.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:47.969+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:47.964 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000003.32.10, , , , , ] 2024-06-15T07:49:47.964 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:52.966 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:52.969+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:52.966 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000003.32.10, , , , , ] 2024-06-15T07:49:52.966 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:49:57.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:49:57.973+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:49:57.969 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000003.32.10, , , , , ] 2024-06-15T07:49:57.969 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:50:02.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:02.973+0000 7f1f21f0ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:02.971 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000003.32.10, , , , , ] 2024-06-15T07:50:02.971 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:50:07.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:07.977+0000 7f1f21f0ba40 1 bucket sync caught up with source: 2024-06-15T07:50:07.973 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:50:07.974 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.29.6, , , , 00000000004.34.10, , , , , ] 2024-06-15T07:50:07.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:07.977+0000 7f1f21f0ba40 0 bucket checkpoint complete 2024-06-15T07:50:08.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:08.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:08.002 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:08.058 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:08.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:08.125+0000 7fc3274c6a40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:08.128 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000004.34.10, , , , , ] 2024-06-15T07:50:08.128 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:50:13.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:13.129+0000 7fc3274c6a40 1 bucket sync caught up with source: 2024-06-15T07:50:13.130 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:50:13.130 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:50:13.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:13.129+0000 7fc3274c6a40 0 bucket checkpoint complete 2024-06-15T07:50:13.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:13.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:13.157 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:13.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:13.259 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:13.256+0000 7f307239ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:13.259 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000010.91.6, , 00000000005.59.11, , , , , ] 2024-06-15T07:50:13.260 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:18.262 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:18.260+0000 7f307239ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:18.262 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000010.91.6, , 00000000005.59.11, , , , , ] 2024-06-15T07:50:18.262 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:23.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:23.264+0000 7f307239ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:23.265 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000010.91.6, , 00000000005.59.11, , , , , ] 2024-06-15T07:50:23.265 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:28.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.264+0000 7f307239ca40 1 bucket sync caught up with source: 2024-06-15T07:50:28.267 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:28.267 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:28.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.264+0000 7f307239ca40 0 bucket checkpoint complete 2024-06-15T07:50:28.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.295 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:28.346 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:28.346 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.388+0000 7f242a954a40 1 bucket sync caught up with source: 2024-06-15T07:50:28.391 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:28.391 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000010.91.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:50:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.388+0000 7f242a954a40 0 bucket checkpoint complete 2024-06-15T07:50:28.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.407 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:28.448 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:28.448 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.488+0000 7ff81a123a40 1 bucket sync caught up with source: 2024-06-15T07:50:28.489 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.22.10, , 00000000003.60.10, , 00000000014.62.10, , , , , ] 2024-06-15T07:50:28.489 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.22.10, , 00000000003.60.10, , 00000000014.62.10, , , , , ] 2024-06-15T07:50:28.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:28.488+0000 7ff81a123a40 0 bucket checkpoint complete 2024-06-15T07:50:28.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:28.504 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:28.555 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:28.555 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:28.602 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:28.604+0000 7f17eddd8a40 1 bucket sync caught up with source: 2024-06-15T07:50:28.602 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.22.10, , 00000000003.60.10, , 00000000014.62.10, , , , , ] 2024-06-15T07:50:28.602 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.22.10, , 00000000003.60.10, , 00000000014.62.10, , , , , ] 2024-06-15T07:50:28.602 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:28.604+0000 7f17eddd8a40 0 bucket checkpoint complete 2024-06-15T07:50:28.617 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone2, test-zone1} 2024-06-15T07:50:28.629 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:28.632 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:28.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:28.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:28.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:28.632 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:28.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:28.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:28.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:28.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:28.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:28.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:28.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:29.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:29.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:29.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:29.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:29.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:29.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone2, test-zone1} 2024-06-15T07:50:29.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone3, test-zone1} 2024-06-15T07:50:29.617 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:29.622 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:29.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:29.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:29.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:29.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:29.627 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:29.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:29.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:29.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:29.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:29.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:29.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:29.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:29.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:29.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:29.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:30.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:30.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:30.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:30.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:30.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:30.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone3, test-zone1} 2024-06-15T07:50:30.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone3, test-zone2} 2024-06-15T07:50:30.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:30.529 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:30.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:30.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:30.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:30.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:30.535 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:30.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:30.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:30.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:30.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:30.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:30.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:30.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:30.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:30.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:31.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:31.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:31.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:31.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:31.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:31.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:31.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone3, test-zone2} 2024-06-15T07:50:31.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.464 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:31.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:31.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:31.560 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:31.564+0000 7fd8bb8d5a40 1 bucket sync caught up with source: 2024-06-15T07:50:31.560 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.28.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:50:31.561 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.28.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:50:31.561 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:31.564+0000 7fd8bb8d5a40 0 bucket checkpoint complete 2024-06-15T07:50:31.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.579 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:31.630 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:31.672+0000 7f68ec7bda40 1 bucket sync caught up with source: 2024-06-15T07:50:31.674 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000010.28.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:50:31.674 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000010.28.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:50:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:31.672+0000 7f68ec7bda40 0 bucket checkpoint complete 2024-06-15T07:50:31.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:31.689 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:31.730 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:31.773 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:31.772+0000 7f0ed649ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:31.773 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000007.71.11, , , , , ] 2024-06-15T07:50:31.774 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:36.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:36.772+0000 7f0ed649ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:36.776 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000007.71.11, , , , , ] 2024-06-15T07:50:36.776 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:41.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:41.776+0000 7f0ed649ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:41.778 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000007.71.11, , , , , ] 2024-06-15T07:50:41.778 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:46.776+0000 7f0ed649ca40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:46.780 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000007.71.11, , , , , ] 2024-06-15T07:50:46.780 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:51.780+0000 7f0ed649ca40 1 bucket sync caught up with source: 2024-06-15T07:50:51.783 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:51.783 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:51.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:51.780+0000 7f0ed649ca40 0 bucket checkpoint complete 2024-06-15T07:50:51.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:51.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:51.807 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:51.902 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:51.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:51.944+0000 7f121c225a40 1 bucket sync caught up with source: 2024-06-15T07:50:51.948 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:51.948 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000010.111.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:50:51.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:51.944+0000 7f121c225a40 0 bucket checkpoint complete 2024-06-15T07:50:51.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:51.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:51.964 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:52.005 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:52.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:52.044+0000 7f13c90bea40 1 bucket sync caught up with source: 2024-06-15T07:50:52.045 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.31.10, , 00000000003.58.10, , 00000000014.72.6, , , , , ] 2024-06-15T07:50:52.045 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.31.10, , 00000000003.58.10, , 00000000014.72.6, , , , , ] 2024-06-15T07:50:52.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:52.044+0000 7f13c90bea40 0 bucket checkpoint complete 2024-06-15T07:50:52.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:52.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:52.060 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:52.112 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:52.112 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:52.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:52.180+0000 7f88d7b16a40 1 bucket sync caught up with source: 2024-06-15T07:50:52.179 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.31.10, , 00000000003.58.10, , 00000000014.72.6, , , , , ] 2024-06-15T07:50:52.179 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.31.10, , 00000000003.58.10, , 00000000014.72.6, , , , , ] 2024-06-15T07:50:52.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:52.180+0000 7f88d7b16a40 0 bucket checkpoint complete 2024-06-15T07:50:52.201 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone2, test-zone1} 2024-06-15T07:50:52.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:52.210 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:52.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:52.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:52.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:52.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:52.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:52.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:52.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:52.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:52.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:52.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:52.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:52.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:52.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:52.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:52.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:52.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:52.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:52.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:52.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:52.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:52.906 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone2, test-zone1} 2024-06-15T07:50:52.906 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone3, test-zone1} 2024-06-15T07:50:52.915 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:52.922 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:52.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:52.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:52.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:52.922 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:52.971 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:52.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:52.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:52.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:52.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:52.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:53.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:53.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:53.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:53.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:53.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:53.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:53.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:53.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:53.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:53.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:54.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone3, test-zone1} 2024-06-15T07:50:54.086 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone3, test-zone2} 2024-06-15T07:50:54.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:54.100 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:54.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:54.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:54.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:54.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:54.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:54.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:54.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:54.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:54.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:54.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:54.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:54.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:54.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:54.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:54.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:54.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:54.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:54.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:54.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:54.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:55.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone3, test-zone2} 2024-06-15T07:50:55.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.059 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.110 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.110 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:55.160+0000 7f871f1cba40 1 bucket sync caught up with source: 2024-06-15T07:50:55.157 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.35.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:50:55.157 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000010.35.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:50:55.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:55.160+0000 7f871f1cba40 0 bucket checkpoint complete 2024-06-15T07:50:55.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.174 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.226 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.227 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.264+0000 7fae357c6a40 1 bucket sync caught up with source: 2024-06-15T07:50:55.268 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000010.35.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:50:55.268 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000010.35.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:50:55.268 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.264+0000 7fae357c6a40 0 bucket checkpoint complete 2024-06-15T07:50:55.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.283 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.324 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.324 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.368+0000 7fcbf4988a40 1 bucket sync caught up with source: 2024-06-15T07:50:55.368 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.32.10, , 00000000010.55.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:50:55.369 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.32.10, , 00000000010.55.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:50:55.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.368+0000 7fcbf4988a40 0 bucket checkpoint complete 2024-06-15T07:50:55.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.384 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.516+0000 7f0cad279a40 1 bucket sync caught up with source: 2024-06-15T07:50:55.517 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.32.10, , 00000000010.55.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:50:55.518 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.32.10, , 00000000010.55.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:50:55.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.516+0000 7f0cad279a40 0 bucket checkpoint complete 2024-06-15T07:50:55.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.533 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.618 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.618 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.656+0000 7f737dc42a40 1 bucket sync caught up with source: 2024-06-15T07:50:55.659 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.27.10, , 00000000003.30.10, , 00000000014.68.10, , , , , ] 2024-06-15T07:50:55.659 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.27.10, , 00000000003.30.10, , 00000000014.68.10, , , , , ] 2024-06-15T07:50:55.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:50:55.656+0000 7f737dc42a40 0 bucket checkpoint complete 2024-06-15T07:50:55.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:55.674 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:55.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:55.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:55.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:55.776+0000 7f40eb0b2a40 1 bucket sync caught up with source: 2024-06-15T07:50:55.772 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.27.10, , 00000000003.30.10, , 00000000014.68.10, , , , , ] 2024-06-15T07:50:55.772 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.27.10, , 00000000003.30.10, , 00000000014.68.10, , , , , ] 2024-06-15T07:50:55.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:55.776+0000 7f40eb0b2a40 0 bucket checkpoint complete 2024-06-15T07:50:55.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone2, test-zone1} 2024-06-15T07:50:55.794 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:55.797 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:55.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:55.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:55.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:55.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:55.803 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:55.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:55.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:55.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:55.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:55.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:55.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:55.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:56.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:56.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:56.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:56.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:56.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:56.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:56.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:56.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:56.751 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone2, test-zone1} 2024-06-15T07:50:56.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone3, test-zone1} 2024-06-15T07:50:56.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:56.765 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:56.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:56.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:56.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:56.765 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:56.771 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:56.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:56.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:56.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:56.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:56.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:56.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:56.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:56.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:57.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:57.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:57.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:57.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:57.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:57.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:57.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:57.750 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone3, test-zone1} 2024-06-15T07:50:57.751 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone3, test-zone2} 2024-06-15T07:50:57.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:50:57.764 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:57.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:57.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:57.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:57.764 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:50:57.768 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:50:57.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:50:57.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:50:57.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:50:57.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:57.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:57.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:50:57.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:57.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:58.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:50:58.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:58.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:58.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:50:58.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:58.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:58.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:50:58.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone3, test-zone2} 2024-06-15T07:50:58.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:58.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:50:58.889 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:50:58.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:50:58.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:50:58.986 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:50:58.988+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:50:58.986 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:50:58.986 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:03.989 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:03.991+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:03.989 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:03.989 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:08.991 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:08.991+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:08.991 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:08.991 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:13.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:13.995+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:13.993 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:13.993 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:18.995 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:18.995+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:18.995 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:18.995 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:23.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:23.999+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:23.997 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:23.998 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:28.999 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:29.003+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:29.000 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:29.000 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:34.002 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:34.003+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:34.002 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:34.002 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:39.005 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:39.007+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:39.005 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:39.005 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:44.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:44.006+0000 7fbc147eaa40 1 waiting for incremental sync to catch up: 2024-06-15T07:51:44.007 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000010.29.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:44.008 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:49.009 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:49.010+0000 7fbc147eaa40 1 bucket sync caught up with source: 2024-06-15T07:51:49.009 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:49.010 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:49.010 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:49.010+0000 7fbc147eaa40 0 bucket checkpoint complete 2024-06-15T07:51:49.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.044 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:49.097 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:49.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.136+0000 7f61ae9b1a40 1 bucket sync caught up with source: 2024-06-15T07:51:49.139 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:49.139 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000011.32.6, , 00000000003.53.10, , 00000000007.40.10, , , , , ] 2024-06-15T07:51:49.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.136+0000 7f61ae9b1a40 0 bucket checkpoint complete 2024-06-15T07:51:49.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.154 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:49.195 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:49.195 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:49.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.236+0000 7f61ab31ba40 1 bucket sync caught up with source: 2024-06-15T07:51:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000011.114.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:51:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000011.114.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:51:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.236+0000 7f61ab31ba40 0 bucket checkpoint complete 2024-06-15T07:51:49.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.254 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:49.336 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:49.337 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:49.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.376+0000 7f9a33adea40 1 bucket sync caught up with source: 2024-06-15T07:51:49.379 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.39.10, , 00000000011.114.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:51:49.379 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.39.10, , 00000000011.114.6, , 00000000007.62.10, , , , , ] 2024-06-15T07:51:49.379 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.376+0000 7f9a33adea40 0 bucket checkpoint complete 2024-06-15T07:51:49.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.395 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:49.477 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:49.477 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:49.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.516+0000 7fedb76fea40 1 bucket sync caught up with source: 2024-06-15T07:51:49.519 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.22.10, , 00000000003.60.10, , 00000000015.65.6, , , , , ] 2024-06-15T07:51:49.519 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.22.10, , 00000000003.60.10, , 00000000015.65.6, , , , , ] 2024-06-15T07:51:49.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:49.516+0000 7fedb76fea40 0 bucket checkpoint complete 2024-06-15T07:51:49.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:49.534 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:49.584 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:49.584 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:49.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:49.630+0000 7fbc3a2e7a40 1 bucket sync caught up with source: 2024-06-15T07:51:49.631 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.22.10, , 00000000003.60.10, , 00000000015.65.6, , , , , ] 2024-06-15T07:51:49.631 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.22.10, , 00000000003.60.10, , 00000000015.65.6, , , , , ] 2024-06-15T07:51:49.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:49.630+0000 7fbc3a2e7a40 0 bucket checkpoint complete 2024-06-15T07:51:49.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone2, test-zone1} 2024-06-15T07:51:49.653 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:49.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:49.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:49.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:49.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:49.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:49.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:49.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:49.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:49.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:49.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:49.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:50.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:50.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:50.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:50.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:50.604 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone2, test-zone1} 2024-06-15T07:51:50.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone3, test-zone1} 2024-06-15T07:51:50.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:50.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:50.622 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:50.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:50.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:50.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:50.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:50.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:50.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:50.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:50.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:50.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:50.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:51.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:51.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:51.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:51.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:51.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:51.539 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone3, test-zone1} 2024-06-15T07:51:51.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-19 zones={test-zone3, test-zone2} 2024-06-15T07:51:51.547 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:51.552 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:51.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:51.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:51.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:51.553 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:51.557 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:51.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:51.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:51.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:51.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:51.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:51.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:51.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:51.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:51.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:52.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:52.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:52.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:52.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:52.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:52.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:52.479 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-19 zones={test-zone3, test-zone2} 2024-06-15T07:51:52.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.480 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:52.532 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:52.532 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:52.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:52.578+0000 7fb19abbba40 1 bucket sync caught up with source: 2024-06-15T07:51:52.579 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000011.31.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:51:52.579 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.31.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:51:52.579 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:52.578+0000 7fb19abbba40 0 bucket checkpoint complete 2024-06-15T07:51:52.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.595 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:52.646 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.688+0000 7efcf829ba40 1 bucket sync caught up with source: 2024-06-15T07:51:52.689 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000011.31.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:51:52.689 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000011.31.6, , 00000000003.51.10, , 00000000007.53.10, , , , , ] 2024-06-15T07:51:52.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.688+0000 7efcf829ba40 0 bucket checkpoint complete 2024-06-15T07:51:52.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.705 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:52.745 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:52.746 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.788+0000 7f304c943a40 1 bucket sync caught up with source: 2024-06-15T07:51:52.788 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000011.116.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:51:52.788 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000011.116.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:51:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.788+0000 7f304c943a40 0 bucket checkpoint complete 2024-06-15T07:51:52.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.803 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:52.889 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:52.889 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:52.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.932+0000 7f6665660a40 1 bucket sync caught up with source: 2024-06-15T07:51:52.934 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.36.10, , 00000000011.116.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:51:52.934 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.36.10, , 00000000011.116.6, , 00000000008.73.10, , , , , ] 2024-06-15T07:51:52.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:52.932+0000 7f6665660a40 0 bucket checkpoint complete 2024-06-15T07:51:52.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:52.950 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:53.034 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:53.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:53.072+0000 7f22666daa40 1 bucket sync caught up with source: 2024-06-15T07:51:53.075 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.31.10, , 00000000003.58.10, , 00000000015.75.6, , , , , ] 2024-06-15T07:51:53.076 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.31.10, , 00000000003.58.10, , 00000000015.75.6, , , , , ] 2024-06-15T07:51:53.076 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:53.072+0000 7f22666daa40 0 bucket checkpoint complete 2024-06-15T07:51:53.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:53.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:53.091 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:53.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:53.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:53.186+0000 7faf027d6a40 1 bucket sync caught up with source: 2024-06-15T07:51:53.187 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.31.10, , 00000000003.58.10, , 00000000015.75.6, , , , , ] 2024-06-15T07:51:53.187 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.31.10, , 00000000003.58.10, , 00000000015.75.6, , , , , ] 2024-06-15T07:51:53.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:53.186+0000 7faf027d6a40 0 bucket checkpoint complete 2024-06-15T07:51:53.202 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone2, test-zone1} 2024-06-15T07:51:53.208 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:53.212 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:53.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:53.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:53.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:53.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:53.217 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:53.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:53.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:53.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:53.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:53.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:53.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:53.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:53.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:53.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:53.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:53.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:53.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:53.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:53.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:53.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:54.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone2, test-zone1} 2024-06-15T07:51:54.140 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone3, test-zone1} 2024-06-15T07:51:54.149 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:54.199 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:54.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:54.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:54.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:54.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:54.204 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:54.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:54.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:54.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:54.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:54.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:54.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:54.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:54.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:54.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:54.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:54.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:54.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:55.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:55.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:55.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:55.383 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone3, test-zone1} 2024-06-15T07:51:55.383 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-20 zones={test-zone3, test-zone2} 2024-06-15T07:51:55.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:55.397 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:55.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:55.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:55.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:55.397 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:55.403 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:55.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:55.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:55.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:55.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:55.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:55.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:55.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:55.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:55.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:55.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:55.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:56.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:56.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:56.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:56.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:56.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-20 zones={test-zone3, test-zone2} 2024-06-15T07:51:56.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.403 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:56.453 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:56.453 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:56.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:56.502+0000 7febceb2ba40 1 bucket sync caught up with source: 2024-06-15T07:51:56.501 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000011.38.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:51:56.501 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000011.38.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:51:56.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:56.502+0000 7febceb2ba40 0 bucket checkpoint complete 2024-06-15T07:51:56.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.517 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:56.570 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.608+0000 7fd5354c4a40 1 bucket sync caught up with source: 2024-06-15T07:51:56.612 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000011.38.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:51:56.612 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000011.38.6, , 00000000003.21.10, , 00000000007.32.10, , , , , ] 2024-06-15T07:51:56.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.608+0000 7fd5354c4a40 0 bucket checkpoint complete 2024-06-15T07:51:56.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.627 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:56.668 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:56.668 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:56.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.708+0000 7f945f15ea40 1 bucket sync caught up with source: 2024-06-15T07:51:56.712 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.32.10, , 00000000011.58.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:51:56.712 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.32.10, , 00000000011.58.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:51:56.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.708+0000 7f945f15ea40 0 bucket checkpoint complete 2024-06-15T07:51:56.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.727 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:56.823 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:56.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.864+0000 7fdd7943fa40 1 bucket sync caught up with source: 2024-06-15T07:51:56.868 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.32.10, , 00000000011.58.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:51:56.868 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.32.10, , 00000000011.58.6, , 00000000007.51.10, , , , , ] 2024-06-15T07:51:56.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.864+0000 7fdd7943fa40 0 bucket checkpoint complete 2024-06-15T07:51:56.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.884 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:56.928 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:56.928 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:56.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.968+0000 7f5279a04a40 1 bucket sync caught up with source: 2024-06-15T07:51:56.970 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, 00000000003.27.10, , 00000000003.30.10, , 00000000015.71.6, , , , , ] 2024-06-15T07:51:56.970 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, 00000000003.27.10, , 00000000003.30.10, , 00000000015.71.6, , , , , ] 2024-06-15T07:51:56.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:51:56.968+0000 7f5279a04a40 0 bucket checkpoint complete 2024-06-15T07:51:56.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:51:56.986 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:51:57.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:57.036 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:57.081 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:57.082+0000 7ff092631a40 1 bucket sync caught up with source: 2024-06-15T07:51:57.081 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, 00000000003.27.10, , 00000000003.30.10, , 00000000015.71.6, , , , , ] 2024-06-15T07:51:57.081 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, 00000000003.27.10, , 00000000003.30.10, , 00000000015.71.6, , , , , ] 2024-06-15T07:51:57.081 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:51:57.082+0000 7ff092631a40 0 bucket checkpoint complete 2024-06-15T07:51:57.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone2, test-zone1} 2024-06-15T07:51:57.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:57.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:57.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:57.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:57.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:57.105 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:57.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:57.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:57.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:57.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:57.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:57.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:57.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:57.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:57.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:57.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:57.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:57.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:57.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:57.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:58.000 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone2, test-zone1} 2024-06-15T07:51:58.000 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone3, test-zone1} 2024-06-15T07:51:58.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:58.014 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:58.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:58.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:58.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:58.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:58.020 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:58.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:58.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:58.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:58.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:58.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:58.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:58.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:58.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:58.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:58.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:58.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:58.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:58.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:58.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:58.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:58.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone3, test-zone1} 2024-06-15T07:51:58.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-21 zones={test-zone3, test-zone2} 2024-06-15T07:51:58.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:51:58.949 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:58.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:58.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:58.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:58.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:51:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:51:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2024-06-15T07:51:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2024-06-15T07:51:58.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2024-06-15T07:51:58.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:58.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:59.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:51:59.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:59.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:59.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2024-06-15T07:51:59.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:59.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:59.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2024-06-15T07:51:59.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:59.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:59.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2024-06-15T07:51:59.780 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-21 zones={test-zone3, test-zone2} 2024-06-15T07:51:59.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-06-15T07:51:59.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:51:59.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:51:59.828 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:51:59.872 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:59.872 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:51:59.915 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:51:59.933 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:51:59.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437919.826516_271.1", 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:51:59.826516Z" 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:51:59.935 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:51:59.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:51:59.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:51:59.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:51:59.937 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:51:59.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:51:59.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:00.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:00.022+0000 7f5c6c810a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:00.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:00.022+0000 7f5c6c810a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:52:00.036 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:00.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:00.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437919.797176_266.1", 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:51:59.797176Z", 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:00.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:00.040 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:52:00Z" 2024-06-15T07:52:00.041 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:52:00.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718437919.826516_271.1 target=1_1718437919.797176_266.1 2024-06-15T07:52:05.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:52:05.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:52:05.044 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:52:05.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:05.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:05.125 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:05.126+0000 7f2ee7cf2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:05.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:05.126+0000 7f2ee7cf2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:05.140 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:52:05.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:05.142 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:05.143 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437919.826516_271.1", 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:51:59.826516Z", 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:52:05.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:52:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:52:05.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:52:05.146 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:52:05Z" 2024-06-15T07:52:05.146 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:52:05.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:52:05.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:52:05.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:52:05.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:52:05.147 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:52:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:05.192 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:05.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:05.224+0000 7fe87d79ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:05.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:05.228+0000 7fe87d79ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.243 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.244 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:52:05.245 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437919.826516_271.1", 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:51:59.826516Z", 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:52:05.246 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:52:05.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:52:05.248 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:52:05Z" 2024-06-15T07:52:05.248 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:52:05.249 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:52:05.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:05.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:05.264 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:52:05.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:05.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:05.358 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:05.358+0000 7fe8cdd5ba40 1 waiting to reach incremental sync.. 2024-06-15T07:52:10.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:10.362+0000 7fe8cdd5ba40 1 bucket sync caught up with source: 2024-06-15T07:52:10.361 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2024-06-15T07:52:10.362 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000002.10.11, , , , , ] 2024-06-15T07:52:10.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:52:10.362+0000 7fe8cdd5ba40 0 bucket checkpoint complete 2024-06-15T07:52:10.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:10.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:10.387 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:52:10.439 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:10.439 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:10.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:10.480+0000 7f8a8fcaca40 1 waiting to reach incremental sync.. 2024-06-15T07:52:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:15.480+0000 7f8a8fcaca40 1 bucket sync caught up with source: 2024-06-15T07:52:15.484 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2024-06-15T07:52:15.484 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000002.10.11, , , , , ] 2024-06-15T07:52:15.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:15.480+0000 7f8a8fcaca40 0 bucket checkpoint complete 2024-06-15T07:52:15.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:15.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:15.511 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:52:15.562 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:15.563 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:15.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:15.608+0000 7f03bbd6ca40 1 waiting to reach incremental sync.. 2024-06-15T07:52:20.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:20.607+0000 7f03bbd6ca40 1 waiting to reach incremental sync.. 2024-06-15T07:52:25.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:25.607+0000 7f03bbd6ca40 1 waiting to reach incremental sync.. 2024-06-15T07:52:30.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:30.611+0000 7f03bbd6ca40 1 bucket sync caught up with source: 2024-06-15T07:52:30.615 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.27.10, , , , , ] 2024-06-15T07:52:30.615 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000002.25.11, , , , , ] 2024-06-15T07:52:30.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:30.611+0000 7f03bbd6ca40 0 bucket checkpoint complete 2024-06-15T07:52:30.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:30.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:30.644 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:52:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:30.698 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:30.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:30.755+0000 7fcbc4de3a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:30.757 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000002.25.11, , , , , ] 2024-06-15T07:52:30.757 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.29.10, , , , , ] 2024-06-15T07:52:35.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:35.759+0000 7fcbc4de3a40 1 bucket sync caught up with source: 2024-06-15T07:52:35.761 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.29.10, , , , , ] 2024-06-15T07:52:35.761 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.29.10, , , , , ] 2024-06-15T07:52:35.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:35.759+0000 7fcbc4de3a40 0 bucket checkpoint complete 2024-06-15T07:52:35.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:35.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:52:35.786 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:52:35.837 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:52:35.838 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:52:35.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:35.875+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:35.880 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:52:35.880 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:52:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:40.879+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:40.882 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:52:40.882 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:52:45.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:45.883+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:45.885 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:52:45.885 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:52:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:50.883+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:50.888 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:52:50.888 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:52:55.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:52:55.887+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:52:55.890 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:52:55.891 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:00.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:53:00.891+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:00.893 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:53:00.893 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:05.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:53:05.891+0000 7fc085c50a40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:05.896 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000003.35.10, , , , , ] 2024-06-15T07:53:05.896 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:10.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:53:10.895+0000 7fc085c50a40 1 bucket sync caught up with source: 2024-06-15T07:53:10.898 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:10.898 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:10.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:53:10.895+0000 7fc085c50a40 0 bucket checkpoint complete 2024-06-15T07:53:10.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:53:10.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:53:10.933 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:53:10.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:53:10.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:53:11.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:11.037+0000 7fe89f757a40 1 bucket sync caught up with source: 2024-06-15T07:53:11.031 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:11.031 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.37.10, , , , , ] 2024-06-15T07:53:11.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:11.037+0000 7fe89f757a40 0 bucket checkpoint complete 2024-06-15T07:53:11.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone2, test-zone1} 2024-06-15T07:53:11.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:53:11.065 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.065 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:53:11.071 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.208 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone2, test-zone1} 2024-06-15T07:53:11.208 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone3, test-zone1} 2024-06-15T07:53:11.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:53:11.222 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:53:11.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.415 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone3, test-zone1} 2024-06-15T07:53:11.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone3, test-zone2} 2024-06-15T07:53:11.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:53:11.475 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:53:11.513 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:53:11.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:53:11.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone3, test-zone2} 2024-06-15T07:53:12.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:53:12.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:53:12.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:53:12.755 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:53:12.755 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:53:12.799 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:12.801+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:12.799 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , , , ] 2024-06-15T07:53:12.799 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:17.802 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:17.804+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:17.802 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , , , ] 2024-06-15T07:53:17.802 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:22.804 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:22.808+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:22.804 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , , , ] 2024-06-15T07:53:22.804 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:27.806 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:27.808+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:27.806 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000007.36.6, , ] 2024-06-15T07:53:27.806 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:32.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:32.812+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:32.809 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000007.36.6, , ] 2024-06-15T07:53:32.809 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:37.812 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:37.812+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:37.812 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000007.36.6, , ] 2024-06-15T07:53:37.812 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:42.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:42.816+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:42.813 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000007.36.6, , ] 2024-06-15T07:53:42.814 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:47.815 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:47.820+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:47.816 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000011.44.6, , ] 2024-06-15T07:53:47.816 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:52.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:52.820+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:52.818 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000011.44.6, , ] 2024-06-15T07:53:52.818 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:53:57.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:53:57.824+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:53:57.820 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000011.44.6, , ] 2024-06-15T07:53:57.820 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:54:02.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:54:02.824+0000 7f5acb59aa40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:02.822 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000011.44.6, , ] 2024-06-15T07:54:02.823 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:54:07.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:54:07.828+0000 7f5acb59aa40 1 bucket sync caught up with source: 2024-06-15T07:54:07.825 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000030.91.6, , ] 2024-06-15T07:54:07.825 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000020.61.12, , ] 2024-06-15T07:54:07.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:54:07.828+0000 7f5acb59aa40 0 bucket checkpoint complete 2024-06-15T07:54:07.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:54:07.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:54:07.852 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:54:07.905 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:54:07.905 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:54:07.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:07.943+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:07.948 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000028.85.6, , ] 2024-06-15T07:54:07.948 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:12.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:12.947+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:12.950 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000036.109.6, , ] 2024-06-15T07:54:12.950 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:17.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:17.947+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:17.952 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000036.109.6, , ] 2024-06-15T07:54:17.952 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:22.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:22.950+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:22.955 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000036.109.6, , ] 2024-06-15T07:54:22.955 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:27.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:27.954+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:27.957 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000036.109.6, , ] 2024-06-15T07:54:27.957 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:32.959 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:32.954+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:32.960 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:32.960 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:37.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:37.958+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:37.962 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:37.962 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:42.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:42.958+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:42.964 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:42.964 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:47.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:47.962+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:47.966 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:47.967 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:52.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:52.966+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:52.970 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:52.970 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:54:57.973 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:54:57.970+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:54:57.973 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:54:57.973 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:02.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:02.970+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:02.978 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:02.978 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:07.977 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:07.974+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:07.978 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:07.978 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:12.980 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:12.974+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:12.980 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:12.980 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:17.982 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:17.978+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:17.982 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:17.983 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:22.982+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:22.985 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:22.985 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:27.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:27.982+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:27.987 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:27.988 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:32.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:32.986+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:32.991 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:32.992 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:37.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:37.990+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:37.993 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:37.994 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:42.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:42.990+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:42.996 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:42.996 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:48.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:47.998+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:48.002 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:48.002 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:53.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:53.002+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:53.005 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:53.005 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:55:58.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:55:58.002+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:55:58.008 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:55:58.008 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:03.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:03.006+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:03.011 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:03.011 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:08.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:08.010+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:08.014 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:08.014 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:13.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:13.010+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:13.016 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:13.016 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:18.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:18.018+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:18.018 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:18.019 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:23.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:23.022+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:23.021 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:23.021 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:28.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:28.021+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:28.023 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:28.024 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:33.026 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:33.025+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:33.027 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:33.027 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:38.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:38.029+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:38.029 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:38.029 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:43.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:43.029+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:43.031 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:43.031 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:48.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:48.033+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:48.034 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:48.034 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:53.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:53.033+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:53.036 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:53.036 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:56:58.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:56:58.037+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:56:58.039 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:56:58.039 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:57:03.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:03.041+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:57:03.041 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:57:03.041 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:57:08.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:08.045+0000 7f87bb845a40 1 waiting for incremental sync to catch up: 2024-06-15T07:57:08.047 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000039.117.6, , ] 2024-06-15T07:57:08.047 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:57:13.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.049+0000 7f87bb845a40 1 bucket sync caught up with source: 2024-06-15T07:57:13.051 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:57:13.051 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.14.10, , , 00000000045.129.10, , ] 2024-06-15T07:57:13.051 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.049+0000 7f87bb845a40 0 bucket checkpoint complete 2024-06-15T07:57:13.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.088 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:57:13.151 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:13.152 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.197+0000 7f7bc544da40 1 bucket sync caught up with source: 2024-06-15T07:57:13.198 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000045.138.10, , ] 2024-06-15T07:57:13.198 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000045.138.10, , ] 2024-06-15T07:57:13.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.197+0000 7f7bc544da40 0 bucket checkpoint complete 2024-06-15T07:57:13.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.222 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:57:13.263 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:13.263 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.305+0000 7fae25d8da40 1 bucket sync caught up with source: 2024-06-15T07:57:13.306 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.29.10, , , 00000000045.138.10, , ] 2024-06-15T07:57:13.306 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.29.10, , , 00000000045.138.10, , ] 2024-06-15T07:57:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.305+0000 7fae25d8da40 0 bucket checkpoint complete 2024-06-15T07:57:13.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:13.321 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:57:13.410 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:13.410 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:13.451 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:13.449+0000 7f424a517a40 1 waiting for incremental sync to catch up: 2024-06-15T07:57:13.451 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.37.10, , , 00000000045.135.10, , ] 2024-06-15T07:57:13.451 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:18.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:18.453+0000 7f424a517a40 1 waiting for incremental sync to catch up: 2024-06-15T07:57:18.453 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.37.10, , , 00000000045.135.10, , ] 2024-06-15T07:57:18.453 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:23.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:23.461+0000 7f424a517a40 1 waiting for incremental sync to catch up: 2024-06-15T07:57:23.462 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.37.10, , , 00000000045.135.10, , ] 2024-06-15T07:57:23.462 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:28.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:28.461+0000 7f424a517a40 1 bucket sync caught up with source: 2024-06-15T07:57:28.465 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:28.465 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:28.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:28.461+0000 7f424a517a40 0 bucket checkpoint complete 2024-06-15T07:57:28.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:28.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:28.495 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:57:28.545 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:28.545 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:28.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:28.596+0000 7f7c9443fa40 1 bucket sync caught up with source: 2024-06-15T07:57:28.591 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:28.591 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , 00000000004.37.10, , , 00000000046.137.10, , ] 2024-06-15T07:57:28.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:28.596+0000 7f7c9443fa40 0 bucket checkpoint complete 2024-06-15T07:57:28.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone2, test-zone1} 2024-06-15T07:57:28.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:57:28.625 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:28.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:28.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:28.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:28.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:28.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:28.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:28.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:28.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:29.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:30.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:31.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:32.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:33.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.424 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone2, test-zone1} 2024-06-15T07:57:34.424 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone3, test-zone1} 2024-06-15T07:57:34.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:57:34.445 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:34.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.448 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:57:34.454 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:34.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:34.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:34.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:34.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:34.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:35.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:36.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:37.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:38.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:39.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:40.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:41.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone3, test-zone1} 2024-06-15T07:57:42.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-22 zones={test-zone3, test-zone2} 2024-06-15T07:57:42.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:57:42.460 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:42.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:57:42.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:42.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:42.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-06-15T07:57:42.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:42.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:43.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:44.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:45.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:46.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:47.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:48.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:49.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:50.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:57:50.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-22 zones={test-zone3, test-zone2} 2024-06-15T07:57:50.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.238 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.283 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.284 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.343 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:49:47.210181Z" 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:54:11.233412Z" 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.344 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:52:43.223311Z" 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:43:55.195522Z" 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.345 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:57:50.346 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-06-15T07:57:50.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.346 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.393 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.393 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.448 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "instance_id": 16115834312843160003 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:50.449 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:49:47.210181Z" 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:50.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:54:11.233412Z" 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:50.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:52:43.223311Z" 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:43:55.195522Z" 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.452 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 0, 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 0 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:57:50Z" 2024-06-15T07:57:50.453 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:57:50.454 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2024-06-15T07:57:50.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.455 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.498 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.498 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:49:47.210181Z" 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.557 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:54:11.233412Z" 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:52:43.223311Z" 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:43:55.195522Z" 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.558 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:57:50.561 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-06-15T07:57:50.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.561 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.640 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "instance_id": 9230054081152360291 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.690 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:49:47.210181Z" 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:57:50.691 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:54:11.233412Z" 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.692 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007562", 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:52:43.223311Z" 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.693 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:43:55.195522Z" 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 0, 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 0 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:57:50Z" 2024-06-15T07:57:50.694 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:57:50.695 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2024-06-15T07:57:50.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.696 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.739 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.739 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout:[ 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:50:58.867784Z" 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-06-15T07:57:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:54:33.133242Z" 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:52:43.120125Z" 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:43:55.094549Z" 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:50.798 INFO:teuthology.orchestra.run.smithi131.stdout:] 2024-06-15T07:57:50.799 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-06-15T07:57:50.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.800 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.847 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.847 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:50.902 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "instance_id": 9204208765021377463 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.903 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:50:58.867784Z" 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.904 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:54:33.133242Z" 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:52:43.120125Z" 2024-06-15T07:57:50.905 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:43:55.094549Z" 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:50.906 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 0, 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 0 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:57:50Z" 2024-06-15T07:57:50.907 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:57:50.908 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2024-06-15T07:57:50.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:50.908 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:50.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:50.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.007 INFO:teuthology.orchestra.run.smithi131.stdout:[ 2024-06-15T07:57:51.007 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.007 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.007 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:50:58.867784Z" 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:54:33.133242Z" 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:52:43.120125Z" 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: "last_update": "2024-06-15T07:43:55.094549Z" 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.008 INFO:teuthology.orchestra.run.smithi131.stdout:] 2024-06-15T07:57:51.009 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-06-15T07:57:51.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.010 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.112 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:57:51.112 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:57:51.112 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "instance_id": 10426822352892935986 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.113 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:50:58.867784Z" 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:54:33.133242Z" 2024-06-15T07:57:51.114 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007280", 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:52:43.120125Z" 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.115 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:43:55.094549Z" 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 0, 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 0 2024-06-15T07:57:51.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.117 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:57:51Z" 2024-06-15T07:57:51.117 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:57:51.117 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2024-06-15T07:57:51.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.118 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.194 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.194 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.252 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:57:51.252 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.252 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-06-15T07:57:51.252 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:49:53.250096Z" 2024-06-15T07:57:51.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:57:12.045313Z" 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:52:49.259866Z" 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:43:39.226878Z" 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.253 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:57:51.254 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-06-15T07:57:51.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.255 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.335 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.387 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:57:51.387 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:57:51.387 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "instance_id": 1548134317917798092 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:57:51.388 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:49:53.250096Z" 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.389 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:57:12.045313Z" 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:57:51.390 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:52:49.259866Z" 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.391 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:43:39.226878Z" 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 0, 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 0 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:57:51Z" 2024-06-15T07:57:51.392 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:57:51.393 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2024-06-15T07:57:51.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.394 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.529 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:57:51.529 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.529 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-06-15T07:57:51.529 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:49:53.250096Z" 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:57:12.045313Z" 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:52:49.259866Z" 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:43:39.226878Z" 2024-06-15T07:57:51.530 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.531 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:57:51.531 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-06-15T07:57:51.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.532 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.578 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.578 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: "instance_id": 5421304209862768881 2024-06-15T07:57:51.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004319", 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:49:53.250096Z" 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.631 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000004178", 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:57:12.045313Z" 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000007421", 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:52:49.259866Z" 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "incremental-sync", 2024-06-15T07:57:51.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "00000000000000000000:00000000000000002914", 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:43:39.226878Z" 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 0, 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 0 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.634 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:57:51Z" 2024-06-15T07:57:51.635 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:57:51.636 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2024-06-15T07:57:51.636 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-06-15T07:57:51.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.670 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.714 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.715 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.755 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:57:51.770 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438271.666650_382.1", 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:57:51.666650Z" 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.771 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:57:51.772 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:57:51.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.773 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.852 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:51.856+0000 7f91bbfcca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:57:51.852 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:51.856+0000 7f91bbfcca40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:57:51.867 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:57:51.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.869 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:57:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:57:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:57:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:57:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438271.666650_382.1", 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:57:51.666650Z", 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:57:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:57:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:57:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:57:51Z" 2024-06-15T07:57:51.878 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:57:51.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:57:51.882 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:57:51.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:57:51.883 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:57:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:51.929 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:51.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:51.965+0000 7f832870aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:57:51.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:57:51.965+0000 7f832870aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:57:51.981 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:57:51.981 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:57:51.982 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:57:51.983 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438271.666650_382.1", 2024-06-15T07:57:51.984 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:57:51.666650Z", 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:57:51.985 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:57:51Z" 2024-06-15T07:57:51.986 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:57:51.987 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:57:51.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:51.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:57:51.997 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:57:52.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:57:52.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:57:52.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:52.096+0000 7f90c7be9a40 1 waiting to reach incremental sync.. 2024-06-15T07:57:57.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:57:57.096+0000 7f90c7be9a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:02.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:02.096+0000 7f90c7be9a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:07.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:07.096+0000 7f90c7be9a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:12.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:12.100+0000 7f90c7be9a40 1 bucket sync caught up with source: 2024-06-15T07:58:12.098 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:12.098 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:12.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:12.100+0000 7f90c7be9a40 0 bucket checkpoint complete 2024-06-15T07:58:12.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:12.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:12.127 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:12.177 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:12.177 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:12.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:12.221+0000 7f8b9ff46a40 1 bucket sync caught up with source: 2024-06-15T07:58:12.223 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:12.223 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:12.223 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:12.221+0000 7f8b9ff46a40 0 bucket checkpoint complete 2024-06-15T07:58:12.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:12.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:12.238 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:12.284 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:12.285 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:12.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:12.357+0000 7f367d545a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:17.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:17.361+0000 7f367d545a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:22.361+0000 7f367d545a40 1 waiting to reach incremental sync.. 2024-06-15T07:58:27.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:27.365+0000 7f367d545a40 1 bucket sync caught up with source: 2024-06-15T07:58:27.365 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.62.6, , ] 2024-06-15T07:58:27.365 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.62.6, , ] 2024-06-15T07:58:27.365 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:27.365+0000 7f367d545a40 0 bucket checkpoint complete 2024-06-15T07:58:27.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:27.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:27.398 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:27.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:27.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:27.493+0000 7f76bf23ba40 1 bucket sync caught up with source: 2024-06-15T07:58:27.494 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.62.6, , ] 2024-06-15T07:58:27.495 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.62.6, , ] 2024-06-15T07:58:27.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:27.493+0000 7f76bf23ba40 0 bucket checkpoint complete 2024-06-15T07:58:27.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:27.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:27.510 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:27.551 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:27.590 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:27.589+0000 7f55d01bba40 1 waiting to reach incremental sync.. 2024-06-15T07:58:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:32.593+0000 7f55d01bba40 1 bucket sync caught up with source: 2024-06-15T07:58:32.593 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.77.6, , ] 2024-06-15T07:58:32.594 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.77.6, , ] 2024-06-15T07:58:32.594 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:32.593+0000 7f55d01bba40 0 bucket checkpoint complete 2024-06-15T07:58:32.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:32.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:32.623 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:32.674 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:32.674 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:32.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:32.723+0000 7f39a50aba40 1 bucket sync caught up with source: 2024-06-15T07:58:32.721 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.77.6, , ] 2024-06-15T07:58:32.721 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.77.6, , ] 2024-06-15T07:58:32.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:32.723+0000 7f39a50aba40 0 bucket checkpoint complete 2024-06-15T07:58:32.736 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T07:58:32.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:58:32.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:32.746 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:58:32.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:32.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:32.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:32.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:32.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T07:58:32.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T07:58:32.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:58:32.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:32.948 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:58:32.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:33.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T07:58:33.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T07:58:33.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:58:33.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:33.159 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:58:33.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:58:33.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:58:33.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T07:58:33.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:33.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:33.364 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:58:33.406 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:33.406 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:33.445 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:58:33.460 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:58:33.460 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:33.460 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:58:33.460 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438313.362205_406.1", 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:58:33.362205Z" 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:33.461 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:58:33.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:58:33.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:33.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:33.463 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:58:33.506 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:33.506 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:33.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:33.543+0000 7ff8be1ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:33.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:33.547+0000 7ff8be1ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:33.560 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:33.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438313.354658_405.1", 2024-06-15T07:58:33.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:58:33.354658Z", 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:33.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:58:33Z" 2024-06-15T07:58:33.565 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:58:33.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438313.362205_406.1 target=1_1718438313.354658_405.1 2024-06-15T07:58:38.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:38.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:38.568 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:58:38.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:38.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:38.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:38.647+0000 7f531b945a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:38.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:38.651+0000 7f531b945a40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:38.661 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:58:38.661 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:58:38.661 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:58:38.661 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:58:38.661 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:58:38.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:58:38.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438313.362205_406.1", 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:58:33.362205Z", 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.664 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:58:38.665 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:58:38Z" 2024-06-15T07:58:38.666 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:58:38.667 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T07:58:38.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T07:58:38.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:38.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:58:38.668 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:58:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:38.712 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:38.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:38.749+0000 7f274cf1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:38.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:58:38.749+0000 7f274cf1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:58:38.765 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T07:58:38.765 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T07:58:38.766 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:38.767 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.768 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438313.362205_406.1", 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:58:33.362205Z", 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.769 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:38.770 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T07:58:38Z" 2024-06-15T07:58:38.771 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T07:58:38.772 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T07:58:38.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:38.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:58:38.823 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:58:38.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:58:38.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:58:38.919 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:38.923+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:58:38.919 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:38.919 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:58:43.922 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:43.923+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:58:43.922 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.38.6, , ] 2024-06-15T07:58:43.922 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:58:48.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:48.927+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:58:48.924 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.40.6, , ] 2024-06-15T07:58:48.924 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:58:53.926 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:53.927+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:58:53.927 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.40.6, , ] 2024-06-15T07:58:53.927 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:58:58.929 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:58:58.930+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:58:58.929 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.40.6, , ] 2024-06-15T07:58:58.929 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:03.931 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:03.930+0000 7f777cb51a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:03.931 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.40.6, , ] 2024-06-15T07:59:03.931 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:08.935 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:08.934+0000 7f777cb51a40 1 bucket sync caught up with source: 2024-06-15T07:59:08.935 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:08.935 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:08.935 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:08.934+0000 7f777cb51a40 0 bucket checkpoint complete 2024-06-15T07:59:08.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:08.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:08.974 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:59:09.024 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:09.025 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:09.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:09.076+0000 7f6810580a40 1 bucket sync caught up with source: 2024-06-15T07:59:09.078 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:09.078 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T07:59:09.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:09.076+0000 7f6810580a40 0 bucket checkpoint complete 2024-06-15T07:59:09.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:09.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:09.094 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:59:09.135 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:09.135 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:09.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:09.176+0000 7fabfde1ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:09.177 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:09.177 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:14.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:14.176+0000 7fabfde1ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:14.180 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:14.180 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:19.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:19.180+0000 7fabfde1ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:19.182 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:19.182 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:24.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:24.180+0000 7fabfde1ba40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:24.184 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:24.184 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:29.184+0000 7fabfde1ba40 1 bucket sync caught up with source: 2024-06-15T07:59:29.187 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:29.187 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:29.184+0000 7fabfde1ba40 0 bucket checkpoint complete 2024-06-15T07:59:29.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:29.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:29.216 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:59:29.314 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:29.314 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:29.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:29.368+0000 7f0852785a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:29.370 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:29.370 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:34.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:34.372+0000 7f0852785a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:34.372 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.67.13, , ] 2024-06-15T07:59:34.373 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:39.372+0000 7f0852785a40 1 bucket sync caught up with source: 2024-06-15T07:59:39.375 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:39.375 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T07:59:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:39.372+0000 7f0852785a40 0 bucket checkpoint complete 2024-06-15T07:59:39.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:39.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:39.408 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T07:59:39.459 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:39.459 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:39.501 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:39.500+0000 7f672fe12a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:39.501 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.82.13, , ] 2024-06-15T07:59:39.501 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:44.504 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:44.500+0000 7f672fe12a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:44.504 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.82.13, , ] 2024-06-15T07:59:44.504 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:49.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:49.504+0000 7f672fe12a40 1 waiting for incremental sync to catch up: 2024-06-15T07:59:49.507 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.82.13, , ] 2024-06-15T07:59:49.507 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:54.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:54.516+0000 7f672fe12a40 1 bucket sync caught up with source: 2024-06-15T07:59:54.518 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:54.518 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:54.519 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T07:59:54.516+0000 7f672fe12a40 0 bucket checkpoint complete 2024-06-15T07:59:54.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:54.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T07:59:54.556 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T07:59:54.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:54.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:54.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:54.669+0000 7fcc6b9e5a40 1 bucket sync caught up with source: 2024-06-15T07:59:54.664 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:54.664 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000004.84.10, , ] 2024-06-15T07:59:54.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:54.669+0000 7fcc6b9e5a40 0 bucket checkpoint complete 2024-06-15T07:59:54.680 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T07:59:54.693 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:59:54.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.696 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:59:54.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:54.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T07:59:54.971 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T07:59:54.981 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:59:54.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:54.986 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:59:55.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.359 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T07:59:55.359 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T07:59:55.368 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T07:59:55.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.415 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T07:59:55.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T07:59:55.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T07:59:55.791 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T07:59:55.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:59:55.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:59:55.803 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:59:55.846 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:55.846 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:55.886 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T07:59:55.901 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438395.802183_437.1", 2024-06-15T07:59:55.902 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:59:55.802183Z" 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T07:59:55.903 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T07:59:55.904 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T07:59:55.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:59:55.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T07:59:55.904 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T07:59:55.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T07:59:55.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T07:59:55.983 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:55.985+0000 7f214205ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:59:55.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T07:59:55.989+0000 7f214205ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T07:59:55.998 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:59:55.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T07:59:56.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438395.794885_436.1", 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:59:56.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:59:55.794885Z", 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T07:59:56.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T07:59:55Z" 2024-06-15T07:59:56.003 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T07:59:56.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438395.802183_437.1 target=1_1718438395.794885_436.1 2024-06-15T08:00:01.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:01.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:01.006 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:01.049 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:01.050 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:01.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:01.093+0000 7f287ad4ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:01.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:01.097+0000 7f287ad4ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:00:01.163 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:00:01.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:01.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438395.802183_437.1", 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:59:55.802183Z", 2024-06-15T08:00:01.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:01.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:00:01Z" 2024-06-15T08:00:01.168 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:00:01.168 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T08:00:01.168 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T08:00:01.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:01.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:01.169 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:01.214 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:01.214 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:01.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:01.252+0000 7f700381aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:01.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:01.252+0000 7f700381aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.270 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:01.271 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:01.272 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438395.802183_437.1", 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:59:55.802183Z", 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:01.273 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:01.274 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T08:00:01Z" 2024-06-15T08:00:01.275 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:00:01.276 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T08:00:01.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:01.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:01.292 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:01.343 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:01.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:01.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:01.393+0000 7f736a068a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:01.388 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T08:00:01.388 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:06.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:06.392+0000 7f736a068a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:06.390 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T08:00:06.390 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:11.392 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:11.396+0000 7f736a068a40 1 bucket sync caught up with source: 2024-06-15T08:00:11.392 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:11.393 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:11.393 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:11.396+0000 7f736a068a40 0 bucket checkpoint complete 2024-06-15T08:00:11.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:11.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:11.420 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:11.472 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:11.473 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:11.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:11.512+0000 7fec37688a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:11.515 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.41.13, , ] 2024-06-15T08:00:11.515 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:16.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:16.516+0000 7fec37688a40 1 bucket sync caught up with source: 2024-06-15T08:00:16.518 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:16.518 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000005.45.18, , ] 2024-06-15T08:00:16.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:16.516+0000 7fec37688a40 0 bucket checkpoint complete 2024-06-15T08:00:16.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:16.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:16.550 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:16.602 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:16.602 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:16.647 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:16.644+0000 7f07462fea40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:16.648 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T08:00:16.648 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000006.75.18, , ] 2024-06-15T08:00:21.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:21.648+0000 7f07462fea40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:21.650 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T08:00:21.650 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000006.75.18, , ] 2024-06-15T08:00:26.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:26.652+0000 7f07462fea40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:26.653 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000004.69.10, , ] 2024-06-15T08:00:26.653 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000006.75.18, , ] 2024-06-15T08:00:31.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:31.652+0000 7f07462fea40 1 bucket sync caught up with source: 2024-06-15T08:00:31.655 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000006.75.18, , ] 2024-06-15T08:00:31.655 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000006.75.18, , ] 2024-06-15T08:00:31.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:31.652+0000 7f07462fea40 0 bucket checkpoint complete 2024-06-15T08:00:31.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:31.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:31.683 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:31.733 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:31.734 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:31.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:31.776+0000 7fe99abe0a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:31.780 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000005.74.7, , ] 2024-06-15T08:00:31.780 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.77.15, , ] 2024-06-15T08:00:36.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:36.780+0000 7fe99abe0a40 1 bucket sync caught up with source: 2024-06-15T08:00:36.783 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000007.77.15, , ] 2024-06-15T08:00:36.783 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.77.15, , ] 2024-06-15T08:00:36.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:36.780+0000 7fe99abe0a40 0 bucket checkpoint complete 2024-06-15T08:00:36.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:36.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:36.807 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:36.858 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:36.858 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:36.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:36.900+0000 7f50fc3c7a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:36.901 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000006.90.18, , ] 2024-06-15T08:00:36.901 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:41.903 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:41.900+0000 7f50fc3c7a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:41.903 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000006.90.18, , ] 2024-06-15T08:00:41.903 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:46.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:46.904+0000 7f50fc3c7a40 1 waiting for incremental sync to catch up: 2024-06-15T08:00:46.905 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000006.90.18, , ] 2024-06-15T08:00:46.906 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:51.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:51.904+0000 7f50fc3c7a40 1 bucket sync caught up with source: 2024-06-15T08:00:51.908 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:51.908 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:51.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:51.904+0000 7f50fc3c7a40 0 bucket checkpoint complete 2024-06-15T08:00:51.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:51.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:51.941 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:51.992 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:51.992 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:52.037 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:52.039+0000 7fad7a1ffa40 1 bucket sync caught up with source: 2024-06-15T08:00:52.037 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:52.037 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000007.92.15, , ] 2024-06-15T08:00:52.037 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:52.039+0000 7fad7a1ffa40 0 bucket checkpoint complete 2024-06-15T08:00:52.052 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T08:00:52.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:00:52.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.066 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:00:52.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.399 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone2, test-zone1} 2024-06-15T08:00:52.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T08:00:52.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:00:52.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.459 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:00:52.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone1} 2024-06-15T08:00:52.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T08:00:52.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:00:52.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:00:52.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:00:52.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:52.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:00:53.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-23 zones={test-zone3, test-zone2} 2024-06-15T08:00:53.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-06-15T08:00:53.016 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-24 2024-06-15T08:00:53.050 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-25 2024-06-15T08:00:53.109 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-26 2024-06-15T08:00:53.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:53.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:53.227 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:53.266 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:53.304 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:00:53.318 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438453.222207_477.1", 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:00:53.222207Z" 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:53.319 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:53.320 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:53.320 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:00:53.320 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:53.320 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:00:53.321 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:00:53.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:53.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:53.321 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:53.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:53.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:53.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:53.403+0000 7f0bb1d2da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:53.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:53.407+0000 7f0bb1d2da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:53.418 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:00:53.418 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:53.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:00:53.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438453.030901_460.1", 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:00:53.030901Z", 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:53.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:00:53.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:00:53Z" 2024-06-15T08:00:53.423 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:00:53.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438453.222207_477.1 target=1_1718438453.030901_460.1 2024-06-15T08:00:58.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:58.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:58.425 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:58.468 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:58.468 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:58.505 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:58.507+0000 7feac4fb3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:58.506 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:58.507+0000 7feac4fb3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:00:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438453.222207_477.1", 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:00:53.222207Z", 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:00:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:00:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:00:58.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:00:58.525 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:00:58Z" 2024-06-15T08:00:58.525 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:00:58.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T08:00:58.526 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T08:00:58.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:58.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:00:58.526 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:00:58.571 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:58.571 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:58.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:58.608+0000 7fd2316faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:58.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:00:58.608+0000 7fd2316faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:00:58.625 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:00:58.625 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T08:00:58.625 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T08:00:58.625 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T08:00:58.625 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T08:00:58.626 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:58.627 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438453.222207_477.1", 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:58.628 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T08:00:53.222207Z", 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:00:58.629 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T08:00:58Z" 2024-06-15T08:00:58.630 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:00:58.631 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T08:00:58.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:58.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:00:58.754 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:00:58.804 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:00:58.804 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:00:58.848 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:00:58.851+0000 7f5c1d0a2a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:03.849 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:03.851+0000 7f5c1d0a2a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:08.850 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:08.851+0000 7f5c1d0a2a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:13.853 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:13.855+0000 7f5c1d0a2a40 1 bucket sync caught up with source: 2024-06-15T08:01:13.853 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2024-06-15T08:01:13.853 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2024-06-15T08:01:13.854 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:13.855+0000 7f5c1d0a2a40 0 bucket checkpoint complete 2024-06-15T08:01:13.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:13.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:13.884 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:13.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:13.980+0000 7f1d8bc0da40 1 bucket sync caught up with source: 2024-06-15T08:01:13.981 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2024-06-15T08:01:13.981 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2024-06-15T08:01:13.981 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:13.980+0000 7f1d8bc0da40 0 bucket checkpoint complete 2024-06-15T08:01:13.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:13.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:13.997 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:14.037 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:14.038 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:14.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:14.080+0000 7f53f2c21a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:19.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:19.080+0000 7f53f2c21a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:24.083 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:24.079+0000 7f53f2c21a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:29.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:29.083+0000 7f53f2c21a40 1 bucket sync caught up with source: 2024-06-15T08:01:29.086 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.61.6, , ] 2024-06-15T08:01:29.087 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.61.6, , ] 2024-06-15T08:01:29.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:29.083+0000 7f53f2c21a40 0 bucket checkpoint complete 2024-06-15T08:01:29.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:29.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:29.116 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:29.206 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:29.206 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:29.251+0000 7f8153e2da40 1 bucket sync caught up with source: 2024-06-15T08:01:29.254 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.61.6, , ] 2024-06-15T08:01:29.254 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.61.6, , ] 2024-06-15T08:01:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:29.251+0000 7f8153e2da40 0 bucket checkpoint complete 2024-06-15T08:01:29.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:29.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:29.270 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:29.312 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:29.312 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:29.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:29.355+0000 7fd2e5c64a40 1 waiting to reach incremental sync.. 2024-06-15T08:01:34.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:34.359+0000 7fd2e5c64a40 1 bucket sync caught up with source: 2024-06-15T08:01:34.364 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.83.6, , ] 2024-06-15T08:01:34.364 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.83.6, , ] 2024-06-15T08:01:34.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:34.359+0000 7fd2e5c64a40 0 bucket checkpoint complete 2024-06-15T08:01:34.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:34.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:34.394 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:34.444 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:34.444 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:34.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:34.490+0000 7f7ab8daea40 1 bucket sync caught up with source: 2024-06-15T08:01:34.489 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.83.6, , ] 2024-06-15T08:01:34.489 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000002.83.6, , ] 2024-06-15T08:01:34.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:34.490+0000 7f7ab8daea40 0 bucket checkpoint complete 2024-06-15T08:01:34.504 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-24 zones={test-zone2, test-zone1} 2024-06-15T08:01:34.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:34.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:34.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-24 zones={test-zone2, test-zone1} 2024-06-15T08:01:34.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-24 zones={test-zone3, test-zone1} 2024-06-15T08:01:34.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:34.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.616 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:34.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-24 zones={test-zone3, test-zone1} 2024-06-15T08:01:34.758 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-24 zones={test-zone3, test-zone2} 2024-06-15T08:01:34.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:34.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:34.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:34.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:35.098 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-24 zones={test-zone3, test-zone2} 2024-06-15T08:01:35.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.099 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:35.149 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:35.149 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:35.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:35.194+0000 7fd2d4dc3a40 1 bucket sync caught up with source: 2024-06-15T08:01:35.194 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:35.194 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:35.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:35.194+0000 7fd2d4dc3a40 0 bucket checkpoint complete 2024-06-15T08:01:35.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.210 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:35.262 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:35.303+0000 7fef8e5c6a40 1 bucket sync caught up with source: 2024-06-15T08:01:35.306 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:35.306 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:35.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:35.303+0000 7fef8e5c6a40 0 bucket checkpoint complete 2024-06-15T08:01:35.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.322 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:35.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:35.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:35.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:35.403+0000 7f0ea2eaba40 1 bucket sync caught up with source: 2024-06-15T08:01:35.405 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2024-06-15T08:01:35.406 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2024-06-15T08:01:35.406 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:35.403+0000 7f0ea2eaba40 0 bucket checkpoint complete 2024-06-15T08:01:35.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:35.421 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:35.511 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:35.554 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:35.551+0000 7f8e63abea40 1 waiting to reach incremental sync.. 2024-06-15T08:01:40.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:40.555+0000 7f8e63abea40 1 bucket sync caught up with source: 2024-06-15T08:01:40.557 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.35.12, , ] 2024-06-15T08:01:40.557 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.35.12, , ] 2024-06-15T08:01:40.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:40.555+0000 7f8e63abea40 0 bucket checkpoint complete 2024-06-15T08:01:40.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:40.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:40.584 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:40.635 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:40.635 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:40.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:40.675+0000 7fd896263a40 1 bucket sync caught up with source: 2024-06-15T08:01:40.678 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-06-15T08:01:40.678 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.36.6, , ] 2024-06-15T08:01:40.678 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:40.675+0000 7fd896263a40 0 bucket checkpoint complete 2024-06-15T08:01:40.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:40.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:40.694 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:40.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:40.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:40.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:40.790+0000 7ff025efaa40 1 bucket sync caught up with source: 2024-06-15T08:01:40.791 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.36.6, , ] 2024-06-15T08:01:40.791 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000002.36.6, , ] 2024-06-15T08:01:40.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:40.790+0000 7ff025efaa40 0 bucket checkpoint complete 2024-06-15T08:01:40.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-25 zones={test-zone2, test-zone1} 2024-06-15T08:01:40.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:40.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:40.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:40.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:40.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-25 zones={test-zone2, test-zone1} 2024-06-15T08:01:40.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-25 zones={test-zone3, test-zone1} 2024-06-15T08:01:40.961 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:40.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:41.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-25 zones={test-zone3, test-zone1} 2024-06-15T08:01:41.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-25 zones={test-zone3, test-zone2} 2024-06-15T08:01:41.118 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:41.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:41.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:41.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-25 zones={test-zone3, test-zone2} 2024-06-15T08:01:41.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.266 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:41.362+0000 7fb2929d5a40 1 bucket sync caught up with source: 2024-06-15T08:01:41.362 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:41.362 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:41.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:41.362+0000 7fb2929d5a40 0 bucket checkpoint complete 2024-06-15T08:01:41.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.378 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.430 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.431 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.471+0000 7fdc2e1aca40 1 bucket sync caught up with source: 2024-06-15T08:01:41.475 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:41.475 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.10.6, , ] 2024-06-15T08:01:41.475 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.471+0000 7fdc2e1aca40 0 bucket checkpoint complete 2024-06-15T08:01:41.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.491 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.532 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.575+0000 7f07968c4a40 1 bucket sync caught up with source: 2024-06-15T08:01:41.577 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.38.6, , ] 2024-06-15T08:01:41.577 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.38.6, , ] 2024-06-15T08:01:41.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.575+0000 7f07968c4a40 0 bucket checkpoint complete 2024-06-15T08:01:41.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.592 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.678 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.679 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.719+0000 7f214f715a40 1 bucket sync caught up with source: 2024-06-15T08:01:41.723 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000002.38.6, , ] 2024-06-15T08:01:41.723 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000002.38.6, , ] 2024-06-15T08:01:41.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.719+0000 7f214f715a40 0 bucket checkpoint complete 2024-06-15T08:01:41.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.739 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.822 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.859+0000 7f1f7bd78a40 1 bucket sync caught up with source: 2024-06-15T08:01:41.864 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2024-06-15T08:01:41.864 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2024-06-15T08:01:41.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:41.859+0000 7f1f7bd78a40 0 bucket checkpoint complete 2024-06-15T08:01:41.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:41.881 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:41.932 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:41.933 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:41.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:41.974+0000 7f9d21683a40 1 bucket sync caught up with source: 2024-06-15T08:01:41.976 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2024-06-15T08:01:41.976 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2024-06-15T08:01:41.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:41.974+0000 7f9d21683a40 0 bucket checkpoint complete 2024-06-15T08:01:41.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-26 zones={test-zone2, test-zone1} 2024-06-15T08:01:42.001 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:42.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:42.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-26 zones={test-zone2, test-zone1} 2024-06-15T08:01:42.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-26 zones={test-zone3, test-zone1} 2024-06-15T08:01:42.145 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:42.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.150 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:42.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-26 zones={test-zone3, test-zone1} 2024-06-15T08:01:42.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-26 zones={test-zone3, test-zone2} 2024-06-15T08:01:42.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:01:42.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:01:42.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:01:42.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:01:42.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-26 zones={test-zone3, test-zone2} 2024-06-15T08:01:42.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-06-15T08:01:42.527 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-27 2024-06-15T08:01:42.563 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-28 2024-06-15T08:01:42.626 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-29 2024-06-15T08:01:42.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:42.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:42.789 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:01:42.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:42.837 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:42.891 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:42.906 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438502.783697_514.1", 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:01:42.783697Z" 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:42.907 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:01:42.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:01:42.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:42.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:42.908 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:01:42.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:42.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:42.988 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:42.990+0000 7fd9798f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:42.989 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:42.990+0000 7fd9798f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:43.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:43.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:01:43.006 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438502.734355_508.1", 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:01:42.734355Z", 2024-06-15T08:01:43.007 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:43.008 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:43.009 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:01:42Z" 2024-06-15T08:01:43.010 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:01:43.011 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438502.783697_514.1 target=1_1718438502.734355_508.1 2024-06-15T08:01:48.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:48.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:48.012 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:01:48.056 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:48.056 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:48.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:48.094+0000 7fe521035a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:48.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:48.094+0000 7fe521035a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:48.108 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:01:48.108 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:01:48.109 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:48.110 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438502.783697_514.1", 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:48.111 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:01:42.783697Z", 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:01:48.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:01:48Z" 2024-06-15T08:01:48.113 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:01:48.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2024-06-15T08:01:48.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2024-06-15T08:01:48.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:48.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:01:48.115 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:01:48.159 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:48.159 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:48.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:48.191+0000 7ff7d0c08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:48.195+0000 7ff7d0c08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": { 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "info": { 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "status": "sync", 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "num_shards": 4, 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:01:48.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "markers": [ 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 0, 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 1, 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:48.212 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 1, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "0.000000", 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 2, 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438502.783697_514.1", 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T08:01:42.783697Z", 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.214 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "key": 3, 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "state": 1, 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "next_step_marker": "", 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "total_entries": 0, 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "pos": 0, 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.215 INFO:teuthology.orchestra.run.smithi012.stdout: "full_sync": { 2024-06-15T08:01:48.216 INFO:teuthology.orchestra.run.smithi012.stdout: "total": 1, 2024-06-15T08:01:48.216 INFO:teuthology.orchestra.run.smithi012.stdout: "complete": 1 2024-06-15T08:01:48.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:01:48.216 INFO:teuthology.orchestra.run.smithi012.stdout: "current_time": "2024-06-15T08:01:48Z" 2024-06-15T08:01:48.216 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:01:48.217 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2024-06-15T08:01:48.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:48.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:48.406 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:48.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:48.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:48.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:48.502+0000 7f495a91fa40 1 waiting to reach incremental sync.. 2024-06-15T08:01:53.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:53.506+0000 7f495a91fa40 1 bucket sync caught up with source: 2024-06-15T08:01:53.504 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.15.7, , ] 2024-06-15T08:01:53.504 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.15.7, , ] 2024-06-15T08:01:53.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:01:53.506+0000 7f495a91fa40 0 bucket checkpoint complete 2024-06-15T08:01:53.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:53.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:53.533 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:53.587 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:53.588 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:53.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:53.627+0000 7f4f5e56ca40 1 waiting to reach incremental sync.. 2024-06-15T08:01:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:58.631+0000 7f4f5e56ca40 1 bucket sync caught up with source: 2024-06-15T08:01:58.634 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.15.7, , ] 2024-06-15T08:01:58.634 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.15.7, , ] 2024-06-15T08:01:58.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:58.631+0000 7f4f5e56ca40 0 bucket checkpoint complete 2024-06-15T08:01:58.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:58.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:01:58.666 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:01:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:01:58.717 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:01:58.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:01:58.759+0000 7fde2d399a40 1 waiting to reach incremental sync.. 2024-06-15T08:02:03.763 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:03.759+0000 7fde2d399a40 1 waiting to reach incremental sync.. 2024-06-15T08:02:08.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:08.763+0000 7fde2d399a40 1 bucket sync caught up with source: 2024-06-15T08:02:08.767 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.34.14, , ] 2024-06-15T08:02:08.767 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.34.14, , ] 2024-06-15T08:02:08.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:08.763+0000 7fde2d399a40 0 bucket checkpoint complete 2024-06-15T08:02:08.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:08.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:08.797 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:08.848 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:08.848 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:08.891+0000 7fce4e7b0a40 1 bucket sync caught up with source: 2024-06-15T08:02:08.893 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.34.14, , ] 2024-06-15T08:02:08.893 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.34.14, , ] 2024-06-15T08:02:08.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:08.891+0000 7fce4e7b0a40 0 bucket checkpoint complete 2024-06-15T08:02:08.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:08.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:08.908 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:08.951 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:08.952 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:08.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:08.991+0000 7f0cb6c5ca40 1 waiting to reach incremental sync.. 2024-06-15T08:02:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:13.991+0000 7f0cb6c5ca40 1 bucket sync caught up with source: 2024-06-15T08:02:13.996 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.36.14, , ] 2024-06-15T08:02:13.996 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.36.14, , ] 2024-06-15T08:02:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:13.991+0000 7f0cb6c5ca40 0 bucket checkpoint complete 2024-06-15T08:02:14.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-27', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.021 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-27 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.072 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.072 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.117 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:14.121+0000 7f4e9f743a40 1 bucket sync caught up with source: 2024-06-15T08:02:14.117 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.36.14, , ] 2024-06-15T08:02:14.117 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.36.14, , ] 2024-06-15T08:02:14.117 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:14.121+0000 7f4e9f743a40 0 bucket checkpoint complete 2024-06-15T08:02:14.135 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-27 zones={test-zone2, test-zone1} 2024-06-15T08:02:14.142 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:14.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:14.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:14.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-27 zones={test-zone2, test-zone1} 2024-06-15T08:02:14.161 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-27 zones={test-zone3, test-zone1} 2024-06-15T08:02:14.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:14.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.219 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:14.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:14.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-27 zones={test-zone3, test-zone1} 2024-06-15T08:02:14.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-27 zones={test-zone3, test-zone2} 2024-06-15T08:02:14.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:14.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:14.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:14.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:14.309 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-27 zones={test-zone3, test-zone2} 2024-06-15T08:02:14.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.309 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.359 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.359 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:14.409+0000 7f0dae780a40 1 bucket sync caught up with source: 2024-06-15T08:02:14.406 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.9.14, , ] 2024-06-15T08:02:14.406 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.9.14, , ] 2024-06-15T08:02:14.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:14.409+0000 7f0dae780a40 0 bucket checkpoint complete 2024-06-15T08:02:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.422 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.475 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.515+0000 7fa9adee2a40 1 bucket sync caught up with source: 2024-06-15T08:02:14.517 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.9.14, , ] 2024-06-15T08:02:14.517 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.9.14, , ] 2024-06-15T08:02:14.517 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.515+0000 7fa9adee2a40 0 bucket checkpoint complete 2024-06-15T08:02:14.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.532 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.572 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.573 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.615+0000 7f31f57a7a40 1 bucket sync caught up with source: 2024-06-15T08:02:14.617 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.47.7, , ] 2024-06-15T08:02:14.617 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.47.7, , ] 2024-06-15T08:02:14.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.615+0000 7f31f57a7a40 0 bucket checkpoint complete 2024-06-15T08:02:14.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.632 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.718 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.719 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.759+0000 7f7f154aba40 1 bucket sync caught up with source: 2024-06-15T08:02:14.764 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.47.7, , ] 2024-06-15T08:02:14.764 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.47.7, , ] 2024-06-15T08:02:14.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.759+0000 7f7f154aba40 0 bucket checkpoint complete 2024-06-15T08:02:14.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.780 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.867 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.867 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:14.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.907+0000 7fcdf3ccba40 1 bucket sync caught up with source: 2024-06-15T08:02:14.909 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.46.14, , ] 2024-06-15T08:02:14.909 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.46.14, , ] 2024-06-15T08:02:14.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:14.907+0000 7fcdf3ccba40 0 bucket checkpoint complete 2024-06-15T08:02:14.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-28', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:14.926 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-28 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:14.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:14.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:15.029+0000 7fcc418d1a40 1 bucket sync caught up with source: 2024-06-15T08:02:15.025 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.46.14, , ] 2024-06-15T08:02:15.025 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.46.14, , ] 2024-06-15T08:02:15.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:15.029+0000 7fcc418d1a40 0 bucket checkpoint complete 2024-06-15T08:02:15.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-28 zones={test-zone2, test-zone1} 2024-06-15T08:02:15.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:15.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.095 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:15.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:15.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-28 zones={test-zone2, test-zone1} 2024-06-15T08:02:15.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-28 zones={test-zone3, test-zone1} 2024-06-15T08:02:15.164 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:15.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.211 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:15.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:15.272 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-28 zones={test-zone3, test-zone1} 2024-06-15T08:02:15.272 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-28 zones={test-zone3, test-zone2} 2024-06-15T08:02:15.280 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:15.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.285 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:15.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:15.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:15.348 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-28 zones={test-zone3, test-zone2} 2024-06-15T08:02:15.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.349 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:15.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:15.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:15.445+0000 7f04dc215a40 1 bucket sync caught up with source: 2024-06-15T08:02:15.444 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000001.32.14, , ] 2024-06-15T08:02:15.444 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000001.32.14, , ] 2024-06-15T08:02:15.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:15.445+0000 7f04dc215a40 0 bucket checkpoint complete 2024-06-15T08:02:15.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.460 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:15.512 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:15.513 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.555+0000 7f0366b6ea40 1 bucket sync caught up with source: 2024-06-15T08:02:15.557 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.32.14, , ] 2024-06-15T08:02:15.557 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.32.14, , ] 2024-06-15T08:02:15.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.555+0000 7f0366b6ea40 0 bucket checkpoint complete 2024-06-15T08:02:15.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.573 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:15.613 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.655+0000 7f2ce996da40 1 bucket sync caught up with source: 2024-06-15T08:02:15.657 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.53.14, , ] 2024-06-15T08:02:15.657 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.53.14, , ] 2024-06-15T08:02:15.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.655+0000 7f2ce996da40 0 bucket checkpoint complete 2024-06-15T08:02:15.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.673 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:15.759 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:15.759 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.803+0000 7f9a8babea40 1 bucket sync caught up with source: 2024-06-15T08:02:15.805 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000001.53.14, , ] 2024-06-15T08:02:15.805 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000001.53.14, , ] 2024-06-15T08:02:15.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.803+0000 7f9a8babea40 0 bucket checkpoint complete 2024-06-15T08:02:15.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.821 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:15.906 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.943+0000 7fed4eb33a40 1 bucket sync caught up with source: 2024-06-15T08:02:15.948 INFO:teuthology.orchestra.run.smithi012.stderr: local status: [, , , , , , , , 00000000003.63.7, , ] 2024-06-15T08:02:15.948 INFO:teuthology.orchestra.run.smithi012.stderr: remote markers: [, , , , , , , , 00000000003.63.7, , ] 2024-06-15T08:02:15.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:02:15.943+0000 7fed4eb33a40 0 bucket checkpoint complete 2024-06-15T08:02:15.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xdvrxv-29', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2024-06-15T08:02:15.964 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xdvrxv-29 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2024-06-15T08:02:16.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:16.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:16.060 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:16.061+0000 7ff6eaf4ca40 1 bucket sync caught up with source: 2024-06-15T08:02:16.060 INFO:teuthology.orchestra.run.smithi131.stderr: local status: [, , , , , , , , 00000000003.63.7, , ] 2024-06-15T08:02:16.060 INFO:teuthology.orchestra.run.smithi131.stderr: remote markers: [, , , , , , , , 00000000003.63.7, , ] 2024-06-15T08:02:16.060 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:16.061+0000 7ff6eaf4ca40 0 bucket checkpoint complete 2024-06-15T08:02:16.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-29 zones={test-zone2, test-zone1} 2024-06-15T08:02:16.082 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:16.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.085 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:16.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:16.100 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-29 zones={test-zone2, test-zone1} 2024-06-15T08:02:16.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-29 zones={test-zone3, test-zone1} 2024-06-15T08:02:16.108 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:16.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:16.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:16.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-29 zones={test-zone3, test-zone1} 2024-06-15T08:02:16.172 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xdvrxv-29 zones={test-zone3, test-zone2} 2024-06-15T08:02:16.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-06-15T08:02:16.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-06-15T08:02:16.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-06-15T08:02:16.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-06-15T08:02:16.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xdvrxv-29 zones={test-zone3, test-zone2} 2024-06-15T08:02:16.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-06-15T08:02:16.250 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-30 2024-06-15T08:02:16.286 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-31 2024-06-15T08:02:16.346 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-32 2024-06-15T08:02:16.554 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-06-15T08:02:16.555 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-33 2024-06-15T08:02:16.593 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-34 2024-06-15T08:02:16.656 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-35 2024-06-15T08:02:16.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-06-15T08:02:16.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-36 2024-06-15T08:02:16.863 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-37 2024-06-15T08:02:16.942 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-38 2024-06-15T08:02:19.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-06-15T08:02:19.091 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-39 2024-06-15T08:02:19.127 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-40 2024-06-15T08:02:19.213 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-41 2024-06-15T08:02:19.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:19.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:19.280 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:19.314 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:19.349 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:02:19.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:02:19.270971Z" 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:02:19.367 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:02:19.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:02:19.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:19.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:19.369 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:19.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:19.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:19.456 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:19.457+0000 7fd48680fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:19.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:19.461+0000 7fd48680fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:19.472 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:19.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:19.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:19.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.477 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:19.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:19.478 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:19.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:19.478 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:19Z" 2024-06-15T08:02:19.478 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:19.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:02:24.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:24.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:24.480 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:24.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:24.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:24.561 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:24.565+0000 7f4073c8aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:24.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:24.565+0000 7f4073c8aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:24.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:24.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:24.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:24.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:24Z" 2024-06-15T08:02:24.582 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:24.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.572026_547.1 2024-06-15T08:02:29.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:29.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:29.585 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:29.628 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:29.628 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:29.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:29.669+0000 7f3d98f93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:29.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:29.669+0000 7f3d98f93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:29.681 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:29Z" 2024-06-15T08:02:29.686 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:29.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:02:34.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:34.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:34.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:34.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:34.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:34.763 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:34.765+0000 7f3cc0848a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:34.763 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:34.765+0000 7f3cc0848a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:34.778 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:34Z" 2024-06-15T08:02:34.783 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:34.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:02:39.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:39.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:39.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:39.829 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:39.829 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:39.866 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:39.869+0000 7f03ced15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:39.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:39.869+0000 7f03ced15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:39.882 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:39.882 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:39.882 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:39.882 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:39Z" 2024-06-15T08:02:39.887 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:39.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:02:44.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:44.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:44.889 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:44.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:44.968 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:44.969+0000 7f61e4e07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:44.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:44.973+0000 7f61e4e07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:44.984 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:44.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:44.986 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:44.987 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.988 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:44.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:44.989 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:44.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:44.989 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:44Z" 2024-06-15T08:02:44.989 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:44.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:02:49.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:49.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:49.991 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:50.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:50.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:50.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:50.072+0000 7f371df97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:50.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:50.072+0000 7f371df97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:50.086 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:50.086 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:50.086 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:50.086 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:50.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:50.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:50.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:50.090 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:50Z" 2024-06-15T08:02:50.091 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:50.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:02:55.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:55.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:02:55.093 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:02:55.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:02:55.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:02:55.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:55.180+0000 7fb75740aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:55.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:02:55.180+0000 7fb75740aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:02:55.194 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:02:55.195 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.196 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:02:55.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:02:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:02:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:02:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:02:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:02:55Z" 2024-06-15T08:02:55.199 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:02:55.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:00.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:00.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:00.201 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:00.242 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:00.242 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:00.277 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:00.280+0000 7f4777c5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:00.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:00.280+0000 7f4777c5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:00.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:00.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:00.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:00.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.297 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:00.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:00.298 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:00.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:00.298 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:00Z" 2024-06-15T08:03:00.298 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:00.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.380042_540.1 2024-06-15T08:03:05.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:05.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:05.301 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:05.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:05.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:05.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:05.380+0000 7f175f934a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:05.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:05.380+0000 7f175f934a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:05.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:05.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:05.392 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:05.392 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:05.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:05.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:05.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:05.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:05Z" 2024-06-15T08:03:05.397 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:05.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:10.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:10.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:10.400 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:10.442 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:10.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:10.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:10.480+0000 7f9bb9e4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:10.480 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:10.484+0000 7f9bb9e4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:10.495 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:10Z" 2024-06-15T08:03:10.500 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:10.501 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:15.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:15.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:15.503 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:15.547 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:15.547 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:15.587 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:15.588+0000 7fad224eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:15.587 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:15.588+0000 7fad224eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:15.602 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:15.603 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:15.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:15.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:15.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:15Z" 2024-06-15T08:03:15.607 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:15.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:20.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:20.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:20.609 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:20.652 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:20.652 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:20.689 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:20.692+0000 7fab47a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:20.690 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:20.692+0000 7fab47a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:20.705 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:20.706 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:20.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:20.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.709 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:20.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:20.710 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:20.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:20.710 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:20Z" 2024-06-15T08:03:20.710 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:20.711 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.742181_561.1 2024-06-15T08:03:25.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:25.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:25.712 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:25.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:25.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:25.792 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:25.792+0000 7fd6d6824a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:25.792 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:25.796+0000 7fd6d6824a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:25.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:25.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:25.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:25.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:25.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:25.813 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:25.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:25.813 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:25Z" 2024-06-15T08:03:25.813 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:25.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:30.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:30.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:30.815 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:30.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:30.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:30.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:30.896+0000 7fa9c1fe8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:30.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:30.900+0000 7fa9c1fe8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:30.914 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:30.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:30.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:30.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:30.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:30Z" 2024-06-15T08:03:30.919 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:30.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:35.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:35.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:35.921 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:35.964 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:35.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:36.001 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:36.003+0000 7fa2ea608a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:36.002 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:36.003+0000 7fa2ea608a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:36.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:36.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:36.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:36.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:36.022 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:36.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:36.023 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:36Z" 2024-06-15T08:03:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:36.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:03:41.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:41.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:41.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:41.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:41.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:41.106 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:41.107+0000 7f84793c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:41.107 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:41.107+0000 7f84793c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:41.122 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:41.122 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:41.122 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:41.122 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:41.122 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:41.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:41.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:41.126 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:41.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:41.127 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:41Z" 2024-06-15T08:03:41.127 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:41.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:46.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:46.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:46.129 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:46.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:46.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:46.210 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:46.211+0000 7fac8dd19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:46.211 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:46.211+0000 7fac8dd19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:46.226 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:46.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:46.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.229 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:46.230 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:46Z" 2024-06-15T08:03:46.231 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:46.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:03:51.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:51.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:51.233 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:51.278 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:51.278 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:51.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:51.315+0000 7fbf270a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:51.316 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:51.319+0000 7fbf270a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:51.331 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:51.331 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:51.331 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:51.331 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:51.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:51.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:51.334 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.500644_546.1", 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.500644Z", 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:51.335 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:51Z" 2024-06-15T08:03:51.336 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:51.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.500644_546.1 2024-06-15T08:03:56.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:56.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:03:56.339 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:03:56.382 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:03:56.382 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:03:56.420 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:56.419+0000 7f824894fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:56.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:03:56.423+0000 7f824894fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:03:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:03:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:03:56Z" 2024-06-15T08:03:56.441 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:03:56.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:04:01.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:01.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:01.444 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:01.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:01.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:01.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:01.523+0000 7f7d6f11fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:01.524 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:01.527+0000 7f7d6f11fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:01.539 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:01.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:04:01.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:01.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:01Z" 2024-06-15T08:04:01.544 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:01.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:04:06.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:06.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:06.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:06.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:06.589 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:06.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:06.627+0000 7f396a005a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:06.626 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:06.627+0000 7f396a005a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:06.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:06.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:06.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:06.645 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:06Z" 2024-06-15T08:04:06.646 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:06.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:04:11.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:11.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:11.649 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:11.692 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:11.692 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:11.729 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:11.731+0000 7fa39d293a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:11.730 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:11.731+0000 7fa39d293a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.745 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:11.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:11.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:11.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:11.749 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:11Z" 2024-06-15T08:04:11.750 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:11.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:04:16.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:16.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:16.752 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:16.794 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:16.794 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:16.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:16.831+0000 7ffad0437a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:16.831 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:16.831+0000 7ffad0437a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:16.845 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:16.845 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:16.845 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:16.845 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:16.845 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:16.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:16.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:16.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:16.849 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:16Z" 2024-06-15T08:04:16.850 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:16.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438538.004281_580.1 2024-06-15T08:04:21.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:21.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:21.853 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:21.897 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:21.897 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:21.935 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:21.935+0000 7ff13dd0da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:21.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:21.935+0000 7ff13dd0da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:21.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:21.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.955 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:21.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:21.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:21.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:21.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:21Z" 2024-06-15T08:04:21.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:21.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438538.004281_580.1 2024-06-15T08:04:26.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:26.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:26.958 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:27.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:27.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:27.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:27.038+0000 7fbea4689a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:27.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:27.038+0000 7fbea4689a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:27.054 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:27.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:27.056 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:27.057 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:27.058 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:27.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:27.060 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:27Z" 2024-06-15T08:04:27.060 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:27.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:04:32.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:32.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:32.062 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:32.104 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:32.104 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:32.141 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:32.142+0000 7f7c55794a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:32.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:32.142+0000 7f7c55794a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.159 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:32.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:32.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:32.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:32.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:32.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:32Z" 2024-06-15T08:04:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:32.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:04:37.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:37.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:37.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:37.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:37.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:37.246 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:37.246+0000 7ff661846a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:37.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:37.246+0000 7ff661846a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:37.262 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:37.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:37.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:37Z" 2024-06-15T08:04:37.268 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:37.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.380042_540.1 2024-06-15T08:04:42.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:42.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:42.271 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:42.314 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:42.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:42.351 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:42.350+0000 7f7621ab9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:42.352 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:42.350+0000 7f7621ab9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:42.367 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:42Z" 2024-06-15T08:04:42.372 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:42.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:04:47.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:47.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:47.374 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:47.419 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:47.419 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:47.455 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:47.454+0000 7f23a8b80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:47.456 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:47.458+0000 7f23a8b80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:47.471 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:47.471 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:47.471 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:47.471 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:47.471 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:47.472 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:47.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:04:47.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:47.475 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:47Z" 2024-06-15T08:04:47.476 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:47.477 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:04:52.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:52.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:52.479 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:52.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:52.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:52.559 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:52.558+0000 7f120195ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:52.560 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:52.558+0000 7f120195ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:52.575 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:52.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:52.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:04:52.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:52.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:52Z" 2024-06-15T08:04:52.580 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:52.581 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:04:57.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:57.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:04:57.583 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:04:57.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:04:57.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:04:57.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:57.666+0000 7f76ba6afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:57.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:04:57.666+0000 7f76ba6afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:04:57.678 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:04:57.678 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:04:57.678 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:04:57.679 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:57.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:04:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:04:57.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:04:57Z" 2024-06-15T08:04:57.683 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:04:57.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:05:02.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:02.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:02.685 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:02.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:02.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:02.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:02.770+0000 7f027aa7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:02.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:02.770+0000 7f027aa7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:02.783 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:02.783 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:02.783 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:02.783 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:02.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:02.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.374557_539.1", 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.374557Z", 2024-06-15T08:05:02.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:02.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:02Z" 2024-06-15T08:05:02.788 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:02.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.374557_539.1 2024-06-15T08:05:07.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:07.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:07.792 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:07.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:07.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:07.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:07.873+0000 7f27aef55a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:07.873 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:07.877+0000 7f27aef55a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:07.887 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:07.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:07.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.631943_554.1", 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.631943Z", 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:07.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:07.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:07Z" 2024-06-15T08:05:07.892 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:07.893 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.631943_554.1 2024-06-15T08:05:12.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:12.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:12.895 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:12.940 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:12.940 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:12.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:12.977+0000 7fa459b35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:12.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:12.981+0000 7fa459b35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:12.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:12.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:12.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:12.997 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:12.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:12.997 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:12Z" 2024-06-15T08:05:12.997 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:12.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:05:17.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:17.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:17.998 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:18.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:18.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:18.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:18.081+0000 7fade03fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:18.079 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:18.081+0000 7fade03fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:18.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:18.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:18.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:18.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:18.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:18.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:18.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:18Z" 2024-06-15T08:05:18.099 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:18.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:05:23.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:23.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:23.102 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:23.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:23.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:23.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:23.181+0000 7fb7170e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:23.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:23.181+0000 7fb7170e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:23.194 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:23.194 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:23.194 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:23.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:23.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:23Z" 2024-06-15T08:05:23.199 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:23.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:05:28.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:28.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:28.201 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:28.246 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:28.246 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:28.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:28.285+0000 7f7c63552a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:28.283 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:28.285+0000 7f7c63552a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:28.297 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:28.298 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:28.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:28.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:28.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:28Z" 2024-06-15T08:05:28.302 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:28.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:05:33.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:33.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:33.305 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:33.372 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:33.372 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:33.414 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:33.417+0000 7f5fcf9a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:33.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:33.417+0000 7f5fcf9a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:33.430 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:33.430 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:33.430 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:33.430 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:33.431 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:33.431 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:33.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:33.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:33.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:33.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:33.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:33.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:33.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:33.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:33.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:33.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:33.434 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:33.434 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:33.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:33.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.492966_545.1", 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.492966Z", 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:33Z" 2024-06-15T08:05:33.441 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:33.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.492966_545.1 2024-06-15T08:05:38.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:38.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:38.444 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:38.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:38.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:38.524 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:38.524+0000 7f07ba3d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:38.525 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:38.528+0000 7f07ba3d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:38.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:38.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:38.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:38Z" 2024-06-15T08:05:38.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:38.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.380042_540.1 2024-06-15T08:05:43.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:43.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:43.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:43.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:43.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:43.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:43.656+0000 7fab723c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:43.657 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:43.660+0000 7fab723c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:43.672 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.677 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:43.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:43.678 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:43.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:43.678 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:43Z" 2024-06-15T08:05:43.678 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:43.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:05:48.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:48.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:48.680 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:48.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:48.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:48.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:48.760+0000 7eff8a226a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:48.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:48.760+0000 7eff8a226a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:48.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:48.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:48Z" 2024-06-15T08:05:48.780 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:48.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:05:53.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:53.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:53.783 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:53.828 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:53.828 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:53.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:53.864+0000 7f4935028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:53.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:53.864+0000 7f4935028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:53.880 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:53Z" 2024-06-15T08:05:53.885 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:53.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:05:58.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:58.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:05:58.887 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:05:58.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:05:58.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:05:58.967 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:58.968+0000 7f83c0739a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:58.968 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:05:58.968+0000 7f83c0739a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:05:58.983 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:05:58.983 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:05:58.983 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:05:58.983 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:05:58.983 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:58.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:58.985 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:58.986 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:05:58.987 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:05:58.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:05:58.989 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:05:58.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:05:58.989 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:05:58Z" 2024-06-15T08:05:58.989 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:05:58.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:03.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:03.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:03.991 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:04.036 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:04.036 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:04.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:04.072+0000 7fa4d31b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:04.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:04.072+0000 7fa4d31b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:04.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:04.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:04.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.492966_545.1", 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.492966Z", 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:04.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:04.092 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:04Z" 2024-06-15T08:06:04.093 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:04.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.492966_545.1 2024-06-15T08:06:09.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:09.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:09.095 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:09.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:09.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:09.203 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:09.203+0000 7ff469a30a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:09.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:09.207+0000 7ff469a30a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:09.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:09.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.492966_545.1", 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.492966Z", 2024-06-15T08:06:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:09Z" 2024-06-15T08:06:09.251 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:09.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.492966_545.1 2024-06-15T08:06:14.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:14.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:14.253 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:14.295 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:14.295 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:14.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:14.331+0000 7f47c6318a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:14.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:14.331+0000 7f47c6318a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:14Z" 2024-06-15T08:06:14.353 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:14.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:19.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:19.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:19.356 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:19.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:19.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:19.435 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:19.435+0000 7fdd46238a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:19.436 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:19.435+0000 7fdd46238a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:19.450 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:19.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:19.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:19.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:19.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:19Z" 2024-06-15T08:06:19.455 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:19.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:24.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:24.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:24.458 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:24.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:24.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:24.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:24.539+0000 7f0a51dbda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:24.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:24.539+0000 7f0a51dbda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:24.556 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:06:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:24.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:24Z" 2024-06-15T08:06:24.561 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:24.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:06:29.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:29.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:29.564 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:29.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:29.617 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:29.653 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:29.651+0000 7ff9cf94ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:29.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:29.655+0000 7ff9cf94ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:29.670 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:29.670 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:29.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:29.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:29.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:29.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:29Z" 2024-06-15T08:06:29.675 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:29.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:34.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:34.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:34.677 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:34.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:34.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:34.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:34.759+0000 7fd838f26a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:34.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:34.759+0000 7fd838f26a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:34.779 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:34.779 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.780 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:34.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:34.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:34.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:34.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:34Z" 2024-06-15T08:06:34.785 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:34.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:39.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:39.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:39.787 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:39.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:39.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:39.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:39.867+0000 7fcb194c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:39.868 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:39.867+0000 7fcb194c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:39.883 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:39.883 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:39.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:39.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:39Z" 2024-06-15T08:06:39.890 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:39.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:06:44.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:44.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:44.892 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:44.934 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:44.934 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:44.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:44.970+0000 7fb6e4869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:44.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:44.974+0000 7fb6e4869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:44.989 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:44.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:44.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:44.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:44.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:44.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:44Z" 2024-06-15T08:06:44.995 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:44.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:06:49.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:49.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:49.998 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:50.041 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:50.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:50.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:50.078+0000 7f6176f2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:50.079 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:50.078+0000 7f6176f2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:50.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:50.099 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:50.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:50.099 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:50Z" 2024-06-15T08:06:50.099 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:50.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:06:55.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:55.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:06:55.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:06:55.145 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:06:55.145 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:06:55.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:55.182+0000 7f1c159f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:55.183 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:06:55.182+0000 7f1c159f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:06:55.198 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:06:55.198 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:55.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:55.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:55.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:06:55.202 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:06:55Z" 2024-06-15T08:06:55.203 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:06:55.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:00.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:00.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:00.206 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:00.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:00.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:00.286 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:00.286+0000 7f358837aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:00.287 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:00.286+0000 7f358837aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:00.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:00.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:00.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:00Z" 2024-06-15T08:07:00.307 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:00.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:05.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:05.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:05.310 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:05.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:05.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:05.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:05.386+0000 7f514637fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:05.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:05.390+0000 7f514637fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:05.406 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:05.406 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:05.406 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:05.406 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:05.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:05.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:05.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:05.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:05Z" 2024-06-15T08:07:05.411 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:05.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:10.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:10.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:10.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:10.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:10.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:10.492 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:10.490+0000 7fb95d0a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:10.493 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:10.490+0000 7fb95d0a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:10Z" 2024-06-15T08:07:10.512 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:10.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:07:15.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:15.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:15.514 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:15.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:15.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:15.599 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:15.602+0000 7fd733544a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:15.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:15.602+0000 7fd733544a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:15.620 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:15.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:15.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:15.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:15.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:15.625 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:15.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:15.625 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:15Z" 2024-06-15T08:07:15.625 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:15.626 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.366541_537.1 2024-06-15T08:07:20.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:20.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:20.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:20.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:20.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:20.709 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:20.709+0000 7f55526a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:20.710 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:20.713+0000 7f55526a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:20.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:20.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:20.727 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:20.728 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:20.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:20Z" 2024-06-15T08:07:20.730 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:20.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438539.270971_594.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:25.732 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-06-15T08:07:25.733 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-42 2024-06-15T08:07:25.773 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-43 2024-06-15T08:07:25.843 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-44 2024-06-15T08:07:26.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:26.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:26.010 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:26.093 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:07:26.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718438845.973139_679.1", 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:07:25.973139Z" 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:07:26.109 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:07:26.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:07:26.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:26.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:26.111 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:26.148 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:26.148 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:26.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:26.185+0000 7f3e34f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:26.183 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:26.185+0000 7f3e34f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:26.197 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:26Z" 2024-06-15T08:07:26.202 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:26.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:07:31.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:31.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:31.205 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:31.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:31.246 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:31.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:31.285+0000 7f5160844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:31.286 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:31.289+0000 7f5160844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:31.302 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:31.302 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:31.302 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:31.302 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:31.302 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:31.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:31.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.582031_549.1", 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.582031Z", 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:31.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:31Z" 2024-06-15T08:07:31.307 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:31.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.582031_549.1 2024-06-15T08:07:36.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:36.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:36.309 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:36.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:36.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:36.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:36.389+0000 7fefadba7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:36.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:36.393+0000 7fefadba7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:36.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:36.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:07:36.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:36.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:36Z" 2024-06-15T08:07:36.410 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:36.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:41.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:41.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:41.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:41.455 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:41.455 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:41.493 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:41.493+0000 7f2197a86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:41.494 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:41.497+0000 7f2197a86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:41.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:41.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:41.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.514 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:41.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:41.515 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:41.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:41.515 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:41Z" 2024-06-15T08:07:41.515 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:41.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:07:46.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:46.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:46.517 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:46.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:46.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:46.594 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:46.593+0000 7f4122402a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:46.594 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:46.597+0000 7f4122402a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:46.611 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:46.611 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:46.611 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:46.611 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:46.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:46.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:07:46.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:46.615 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:46.616 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:46Z" 2024-06-15T08:07:46.617 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:46.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.380042_540.1 2024-06-15T08:07:51.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:51.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:51.618 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:51.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:51.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:51.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:51.709+0000 7feebfa91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:51.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:51.709+0000 7feebfa91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.724 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.726 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.727 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:51.728 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:51Z" 2024-06-15T08:07:51.729 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:51.730 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:07:56.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:56.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:07:56.732 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:07:56.775 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:07:56.775 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:07:56.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:56.813+0000 7ffa3a8bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:56.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:07:56.813+0000 7ffa3a8bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:07:56.829 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:07:56.829 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:56.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:56.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:56.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:56.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.410011_542.1", 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.410011Z", 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:07:56.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:07:56.835 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:07:56Z" 2024-06-15T08:07:56.836 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:07:56.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.410011_542.1 2024-06-15T08:08:01.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:01.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:01.838 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:01.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:01.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:01.918 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:01.920+0000 7f7a27399a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:01.919 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:01.920+0000 7f7a27399a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:01.934 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:01.934 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:01.934 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:01.934 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:01.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:01.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:01.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:01.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:01Z" 2024-06-15T08:08:01.939 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:01.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:08:06.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:06.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:06.941 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:06.985 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:07.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:07.024+0000 7fa0589faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:07.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:07.024+0000 7fa0589faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:07.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:07.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:07.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:07.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:08:07.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:07.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:07Z" 2024-06-15T08:08:07.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:07.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:08:12.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:12.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:12.048 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:12.090 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:12.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:12.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:12.128+0000 7fd541beea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:12.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:12.128+0000 7fd541beea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:12.144 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:12.144 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:12.144 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:12.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:12Z" 2024-06-15T08:08:12.150 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:12.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:08:17.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:17.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:17.152 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:17.196 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:17.196 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:17.244 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:17.244+0000 7f8792f00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:17.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:17.244+0000 7f8792f00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:17.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:17.266 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:17Z" 2024-06-15T08:08:17.266 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:17.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:08:22.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:22.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:22.269 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:22.312 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:22.312 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:22.349 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:22.348+0000 7ff6abfc4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:22.350 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:22.352+0000 7ff6abfc4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:22.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:22.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:22Z" 2024-06-15T08:08:22.370 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:22.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.380042_540.1 2024-06-15T08:08:27.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:27.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:27.373 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:27.416 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:27.417 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:27.458 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:27.456+0000 7fbf732d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:27.459 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:27.460+0000 7fbf732d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:27.474 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:27.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:27.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:27.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:27.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:27Z" 2024-06-15T08:08:27.479 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:27.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:08:32.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:32.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:32.481 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:32.531 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:32.531 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:32.574 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:32.576+0000 7f036b808a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:32.576 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:32.576+0000 7f036b808a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:32.591 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:32.591 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:32.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:32.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:32.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.374557_539.1", 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:32.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.374557Z", 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:32.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:32Z" 2024-06-15T08:08:32.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:32.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.374557_539.1 2024-06-15T08:08:37.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:37.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:37.600 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:37.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:37.644 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:37.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:37.679+0000 7f4012a70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:37.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:37.679+0000 7f4012a70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:37.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:37.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:37.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:37.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:37.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:37.701 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:37.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:37Z" 2024-06-15T08:08:37.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:37.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.959215_574.1 2024-06-15T08:08:42.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:42.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:42.703 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:42.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:42.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:42.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:42.783+0000 7f4fcf9ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:42.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:42.783+0000 7f4fcf9ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:42.801 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:42.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:42.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.582031_549.1", 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:42.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.582031Z", 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:42.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:42Z" 2024-06-15T08:08:42.806 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:42.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.582031_549.1 2024-06-15T08:08:47.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:47.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:47.809 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:47.851 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:47.851 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:47.887 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:47.887+0000 7f9b4a5d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:47.888 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:47.887+0000 7f9b4a5d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:47.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:47.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:47.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:47.908 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:47.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:47.909 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:47Z" 2024-06-15T08:08:47.909 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:47.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:08:52.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:52.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:52.910 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:52.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:52.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:52.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:52.991+0000 7feaa2ed6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:52.994 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:52.991+0000 7feaa2ed6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:53.009 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:53.009 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:53.009 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:53.009 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:53.010 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:53.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:08:53.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:53.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:52Z" 2024-06-15T08:08:53.014 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:53.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.742181_561.1 2024-06-15T08:08:58.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:58.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:08:58.016 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:08:58.060 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:08:58.060 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:08:58.096 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:58.095+0000 7f501bc4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:58.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:08:58.095+0000 7f501bc4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:08:58.112 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:08:58.112 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:58.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:58.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:08:58.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:08:58.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:08:58.118 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:08:58Z" 2024-06-15T08:08:58.118 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:08:58.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:03.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:03.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:03.120 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:03.173 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:03.173 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:03.208 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:03.207+0000 7f6ab41ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:03.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:03.207+0000 7f6ab41ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:03.223 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:03.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:03.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:03.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:03.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:03.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:03.229 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:03Z" 2024-06-15T08:09:03.229 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:03.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:08.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:08.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:08.231 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:08.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:08.276 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:08.313 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:08.311+0000 7f32a10efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:08.314 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:08.311+0000 7f32a10efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:08.329 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:08.329 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:08.329 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:08.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:08.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:08.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:08.333 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:08Z" 2024-06-15T08:09:08.334 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:08.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:13.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:13.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:13.336 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:13.380 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:13.380 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:13.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:13.415+0000 7fec3cb90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:13.419 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:13.419+0000 7fec3cb90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:13.434 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:13.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:13.440 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:13.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:13.440 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:13Z" 2024-06-15T08:09:13.440 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:13.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:18.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:18.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:18.442 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:18.485 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:18.485 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:18.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:18.518+0000 7f4798038a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:18.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:18.522+0000 7f4798038a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:18.539 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:18.539 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:18.539 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:18.539 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:18.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:18.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:18Z" 2024-06-15T08:09:18.544 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:18.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:23.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:23.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:23.546 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:23.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:23.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:23.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:23.622+0000 7f6367ce7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:23.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:23.622+0000 7f6367ce7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:23.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:23.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:23.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:23Z" 2024-06-15T08:09:23.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:23.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:09:28.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:28.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:28.647 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:28.689 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:28.689 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:28.726 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:28.722+0000 7fcc782f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:28.727 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:28.726+0000 7fcc782f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:28.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.744 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.745 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:28.746 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:28Z" 2024-06-15T08:09:28.748 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:28.749 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:09:33.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:33.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:33.750 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:33.793 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:33.793 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:33.831 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:33.830+0000 7f01e2b01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:33.832 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:33.830+0000 7f01e2b01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:33.847 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:33Z" 2024-06-15T08:09:33.852 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:33.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:38.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:38.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:38.854 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:38.904 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:38.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:38.947 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:38.946+0000 7fcd3682da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:38.948 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:38.946+0000 7fcd3682da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:38.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:38.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.967 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:38.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:38.968 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:38.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:38.968 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:38Z" 2024-06-15T08:09:38.968 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:38.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:43.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:43.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:43.970 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:44.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:44.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:44.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:44.050+0000 7fde4ee47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:44.052 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:44.050+0000 7fde4ee47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.068 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:44.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:44.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:44.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:44.072 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:44Z" 2024-06-15T08:09:44.073 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:44.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:09:49.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:49.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:49.076 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:49.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:49.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:49.156 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:49.158+0000 7f3013675a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:49.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:49.158+0000 7f3013675a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:49.172 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:49.172 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:49.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:49.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:49.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.492966_545.1", 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.492966Z", 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:49.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:49.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:49Z" 2024-06-15T08:09:49.178 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:49.179 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.492966_545.1 2024-06-15T08:09:54.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:54.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:54.181 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:54.225 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:54.225 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:54.262 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:54.262+0000 7fa75aaf9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:54.263 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:54.266+0000 7fa75aaf9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:54.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:54.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.410011_542.1", 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.410011Z", 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:54.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:54.283 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:54Z" 2024-06-15T08:09:54.284 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:54.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.410011_542.1 2024-06-15T08:09:59.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:59.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:09:59.286 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:09:59.330 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:09:59.331 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:09:59.367 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:59.369+0000 7fc9a75e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:59.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:09:59.369+0000 7fc9a75e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:09:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:09:59Z" 2024-06-15T08:09:59.389 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:09:59.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:04.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:04.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:04.391 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:04.434 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:04.434 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:04.471 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:04.473+0000 7f2e860d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:04.472 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:04.473+0000 7f2e860d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:04.488 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:04.488 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:04.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:04.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:04.491 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:04.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:04Z" 2024-06-15T08:10:04.493 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:04.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:09.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:09.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:09.496 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:09.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:09.539 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:09.576 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:09.577+0000 7f7969666a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:09.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:09.577+0000 7f7969666a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:09.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:09.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:09.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:09.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:09Z" 2024-06-15T08:10:09.598 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:09.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.380042_540.1 2024-06-15T08:10:14.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:14.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:14.600 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:14.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:14.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:14.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:14.681+0000 7f9c95e3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:14.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:14.681+0000 7f9c95e3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:14.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:14.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:14.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:14.703 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:14.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:14.703 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:14Z" 2024-06-15T08:10:14.703 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:14.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:10:19.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:19.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:19.705 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:19.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:19.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:19.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:19.785+0000 7fbb22996a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:19.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:19.785+0000 7fbb22996a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:19.801 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:19.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:19.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:19.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:19.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:19Z" 2024-06-15T08:10:19.806 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:19.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:24.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:24.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:24.809 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:24.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:24.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:24.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:24.893+0000 7f3ecb715a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:24.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:24.893+0000 7f3ecb715a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:24.908 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:24.909 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:24.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:24.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:24.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:24Z" 2024-06-15T08:10:24.913 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:24.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:29.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:29.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:29.916 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:29.960 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:29.960 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:29.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:29.997+0000 7f55c12b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:29.999 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:30.001+0000 7f55c12b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:30.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:30.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:30.018 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.374557_539.1", 2024-06-15T08:10:30.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.374557Z", 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:30.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:30Z" 2024-06-15T08:10:30.022 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:30.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.374557_539.1 2024-06-15T08:10:35.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:35.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:35.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:35.067 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:35.067 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:35.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:35.117+0000 7f446d6bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:35.117 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:35.117+0000 7f446d6bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.137 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.308115_533.1", 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.308115Z", 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:35.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:35.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:35.142 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:35Z" 2024-06-15T08:10:35.142 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:35.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.308115_533.1 2024-06-15T08:10:40.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:40.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:40.144 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:40.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:40.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:40.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:40.228+0000 7f11fb215a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:40.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:40.228+0000 7f11fb215a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:40.243 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:40.243 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:40.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:40.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:40.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:40.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:40Z" 2024-06-15T08:10:40.248 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:40.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:45.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:45.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:45.251 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:45.296 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:45.296 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:45.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:45.332+0000 7f208aee8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:45.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:45.332+0000 7f208aee8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:45.349 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:45.349 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:45.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:45.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:45.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:45.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:45Z" 2024-06-15T08:10:45.354 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:45.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:10:50.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:50.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:50.356 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:50.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:50.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:50.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:50.436+0000 7fc4bdd91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:50.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:50.440+0000 7fc4bdd91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:50.455 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:50.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:50.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:10:50.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:50.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:50Z" 2024-06-15T08:10:50.460 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:50.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:10:55.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:55.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:10:55.463 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:10:55.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:10:55.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:10:55.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:55.544+0000 7ff42612ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:55.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:10:55.544+0000 7ff42612ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:10:55.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:10:55.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:10:55.567 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:10:55Z" 2024-06-15T08:10:55.568 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:10:55.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:00.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:00.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:00.570 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:00.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:00.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:00.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:00.648+0000 7fee194f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:00.651 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:00.652+0000 7fee194f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:00.667 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:00.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:00.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.374557_539.1", 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:00.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.374557Z", 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:00.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:00Z" 2024-06-15T08:11:00.672 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:00.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.374557_539.1 2024-06-15T08:11:05.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:05.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:05.675 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:05.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:05.727 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:05.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:05.764+0000 7f1108deda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:05.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:05.764+0000 7f1108deda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:05.781 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:05.781 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:05.781 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:05.781 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:05.781 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:05Z" 2024-06-15T08:11:05.786 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:05.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:10.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:10.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:10.789 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:10.833 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:10.833 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:10.870 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:10.868+0000 7ff1850f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:10.871 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:10.872+0000 7ff1850f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:10.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:10.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:11:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:10.896 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:10Z" 2024-06-15T08:11:10.897 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:10.899 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:11:15.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:15.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:15.901 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:15.944 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:15.945 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:15.995 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:15.996+0000 7fa41a436a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:15.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:15.996+0000 7fa41a436a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:16.012 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:16.012 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:16.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:16.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:16.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:16.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:16Z" 2024-06-15T08:11:16.017 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:16.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:21.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:21.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:21.019 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:21.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:21.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:21.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:21.100+0000 7fcc20a31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:21.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:21.100+0000 7fcc20a31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:21.119 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:21.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:21.124 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:21Z" 2024-06-15T08:11:21.124 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:21.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:26.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:26.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:26.126 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:26.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:26.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:26.205 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:26.203+0000 7f9161a5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:26.206 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:26.203+0000 7f9161a5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:26.223 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:26.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:26.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:26.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:26.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:26.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:26Z" 2024-06-15T08:11:26.228 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:26.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.380042_540.1 2024-06-15T08:11:31.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:31.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:31.230 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:31.276 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:31.276 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:31.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:31.311+0000 7f651526da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:31.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:31.315+0000 7f651526da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:31.331 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:31.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:31.337 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:31.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:31.337 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:31Z" 2024-06-15T08:11:31.337 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:31.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:36.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:36.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:36.339 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:36.381 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:36.381 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:36.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:36.415+0000 7f9c0cc81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:36.419 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:36.415+0000 7f9c0cc81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:36.434 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:36.434 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:36.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:36.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:36.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:36.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:36.439 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:36.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:36.439 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:36Z" 2024-06-15T08:11:36.439 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:36.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:41.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:41.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:41.441 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:41.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:41.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:41.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:41.519+0000 7f199fc5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:41.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:41.523+0000 7f199fc5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:41.539 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:41Z" 2024-06-15T08:11:41.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:41.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:11:46.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:46.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:46.548 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:46.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:46.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:46.627 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:46.627+0000 7fddf5790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:46.628 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:46.627+0000 7fddf5790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:46.644 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:46.645 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:46.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:46.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:46.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:46Z" 2024-06-15T08:11:46.649 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:46.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:11:51.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:51.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:51.652 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:51.695 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:51.695 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:51.733 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:51.731+0000 7fb05c08ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:51.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:51.731+0000 7fb05c08ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:51.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.751 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:51.752 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:51.753 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:51.754 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:51.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:51.755 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:51Z" 2024-06-15T08:11:51.755 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:51.756 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:11:56.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:56.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:11:56.757 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:11:56.801 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:11:56.801 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:11:56.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:56.839+0000 7fb1070a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:56.840 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:11:56.839+0000 7fb1070a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:11:56.856 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:11:56.856 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:11:56.856 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:11:56.856 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:11:56.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:56.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.734016_560.1", 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.734016Z", 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:11:56.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:11:56Z" 2024-06-15T08:11:56.861 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:11:56.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.734016_560.1 2024-06-15T08:12:01.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:01.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:01.863 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:01.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:01.946 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:01.943+0000 7f4063684a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:01.947 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:01.947+0000 7f4063684a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:01.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:01.963 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:01.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:01.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:01.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:01.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:01Z" 2024-06-15T08:12:01.968 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:01.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.380042_540.1 2024-06-15T08:12:06.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:06.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:06.971 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:07.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:07.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:07.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:07.051+0000 7f81e37d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:07.052 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:07.051+0000 7f81e37d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:07.068 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:07.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:07.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:07.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:07.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:07Z" 2024-06-15T08:12:07.073 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:07.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:12:12.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:12.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:12.075 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:12.131 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:12.131 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:12.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:12.194+0000 7f01cbc0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:12.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:12.198+0000 7f01cbc0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:12.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:12.215 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:12.215 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:12.215 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:12.215 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:12.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.219 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:12.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:12.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:12.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:12.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:12Z" 2024-06-15T08:12:12.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:12.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:12:17.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:17.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:17.222 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:17.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:17.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:17.304 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:17.302+0000 7fa504ef5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:17.305 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:17.302+0000 7fa504ef5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:17.321 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:17.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:17.321 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:17.321 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:17.322 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:17.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:17.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.325 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:17.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:17.326 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:17.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:17.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:17Z" 2024-06-15T08:12:17.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:17.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:12:22.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:22.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:22.328 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:22.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:22.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:22.409 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:22.406+0000 7fc86b374a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:22.410 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:22.406+0000 7fc86b374a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:22.425 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:22.425 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:22.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:22.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:22.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:22.429 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:22Z" 2024-06-15T08:12:22.430 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:22.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.366541_537.1 2024-06-15T08:12:27.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:27.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:27.432 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:27.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:27.477 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:27.514 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:27.510+0000 7fe601ddda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:27.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:27.514+0000 7fe601ddda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:27.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:27.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:27.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:27.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:27.536 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:27Z" 2024-06-15T08:12:27.536 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:27.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718438845.973139_679.1 target=1_1718438536.403800_541.1 2024-06-15T08:12:32.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2024-06-15T08:12:32.539 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-45 2024-06-15T08:12:32.627 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-46 2024-06-15T08:12:32.790 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-47 2024-06-15T08:12:32.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:32.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:32.890 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:32.924 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:32.924 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:32.963 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:12:32.980 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718439152.829266_756.1", 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:12:32.829266Z" 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:12:32.981 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:12:32.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:12:32.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:32.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:32.983 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:33.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:33.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:33.063 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:33.062+0000 7fe7260f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:33.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:33.062+0000 7fe7260f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:33.081 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:33.082 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:33.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:33.085 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:33Z" 2024-06-15T08:12:33.086 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:33.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:12:38.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:38.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:38.089 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:38.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:38.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:38.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:38.206+0000 7fbba744ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:38.208 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:38.206+0000 7fbba744ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:38.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:38.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.462922_543.1", 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.462922Z", 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:38.228 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:38.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:38.230 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:38.230 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:38.230 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:38Z" 2024-06-15T08:12:38.230 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:38.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.462922_543.1 2024-06-15T08:12:43.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:43.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:43.232 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:43.279 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:43.279 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:43.317 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:43.314+0000 7fc96ea41a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:43.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:43.314+0000 7fc96ea41a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:43.334 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:43.334 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:43.334 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:43.334 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:43.335 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:43.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:43.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:43.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:43Z" 2024-06-15T08:12:43.339 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:43.340 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:12:48.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:48.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:48.342 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:48.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:48.391 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:48.471 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:48.470+0000 7f761bbada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:48.472 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:48.474+0000 7f761bbada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:48.487 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:12:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:48Z" 2024-06-15T08:12:48.492 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:48.493 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:12:49.445 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T08:12:49.446+0000 7fa2e643c640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:12:49.445 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T08:12:49.446+0000 7fa2ed44a640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:12:53.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:53.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:53.494 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:53.537 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:53.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:53.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:53.577+0000 7f9f2de60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:53.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:53.577+0000 7f9f2de60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:53.594 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:53.594 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:53.594 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:53.594 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:53.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:53.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.598 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:53.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:53.599 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:53.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:53.599 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:53Z" 2024-06-15T08:12:53.599 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:53.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:12:58.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:58.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:12:58.601 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:12:58.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:12:58.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:12:58.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:58.681+0000 7f8d95df3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:58.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:12:58.681+0000 7f8d95df3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:12:58.696 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:12:58.696 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:58.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:58.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:58.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:12:58.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:12:58Z" 2024-06-15T08:12:58.701 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:12:58.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.615034_551.1 2024-06-15T08:13:03.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:03.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:03.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:03.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:03.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:03.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:03.785+0000 7fc2e1e98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:03.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:03.785+0000 7fc2e1e98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:03.801 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:03.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:03.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:03.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:03.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:03.806 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:03.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:03.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:03Z" 2024-06-15T08:13:03.807 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:03.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:13:08.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:08.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:08.809 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:08.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:08.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:08.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:08.889+0000 7f63d662fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:08.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:08.893+0000 7f63d662fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:08.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:08.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:08.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:08Z" 2024-06-15T08:13:08.912 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:08.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:13:13.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:13.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:13.915 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:13.957 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:13.958 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:14.010 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:14.009+0000 7fc7f07cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:14.011 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:14.009+0000 7fc7f07cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:14.031 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:14.031 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:14.031 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:14.031 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:14.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:14.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:14.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:14.035 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:14Z" 2024-06-15T08:13:14.036 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:14.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:13:19.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:19.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:19.039 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:19.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:19.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:19.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:19.121+0000 7f55161dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:19.122 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:19.121+0000 7f55161dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:19.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:19.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:19.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T08:13:19.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:19.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:19Z" 2024-06-15T08:13:19.143 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:19.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.955581_573.1 2024-06-15T08:13:24.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:24.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:24.146 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:24.199 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:24.200 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:24.261 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:24.261+0000 7f8888859a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:24.261 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:24.261+0000 7f8888859a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:24.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:24.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:24.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:24.282 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:24Z" 2024-06-15T08:13:24.283 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:24.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:29.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:29.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:29.286 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:29.330 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:29.330 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:29.375 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:29.377+0000 7fa898803a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:29.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:29.381+0000 7fa898803a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:29.399 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:29.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:29.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:29Z" 2024-06-15T08:13:29.404 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:29.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:34.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:34.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:34.407 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:34.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:34.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:34.488 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:34.489+0000 7f4bc7717a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:34.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:34.489+0000 7f4bc7717a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:34.505 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:34.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:34.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:34.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.509 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:34.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:34.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:34.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:34.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:34Z" 2024-06-15T08:13:34.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:34.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:39.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:39.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:39.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:39.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:39.617+0000 7f29f10aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:39.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:39.621+0000 7f29f10aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:39.634 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:39Z" 2024-06-15T08:13:39.639 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:39.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:44.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:44.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:44.684 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:44.684 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:44.722 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:44.720+0000 7ffbd7c8fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:44.723 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:44.720+0000 7ffbd7c8fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:44.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:44.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:44.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:44.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:44.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:44.744 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:44Z" 2024-06-15T08:13:44.744 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:44.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:49.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:49.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:49.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:49.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:49.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:49.826 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:49.824+0000 7f0c84a71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:49.827 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:49.828+0000 7f0c84a71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:49.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:49.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:49.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:49.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:49.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:49.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:49.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:49.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:49Z" 2024-06-15T08:13:49.849 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:49.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:54.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:54.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:54.852 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:13:54.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:13:54.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:13:54.933 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:54.932+0000 7f785b243a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:54.933 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:13:54.932+0000 7f785b243a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:13:54.950 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:13:54.950 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:13:54.950 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:13:54.950 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:13:54.950 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:54.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:54.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:13:54.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:13:54.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:13:54.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:13:54Z" 2024-06-15T08:13:54.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:13:54.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:13:59.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:59.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:13:59.958 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:00.002 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:00.002 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:00.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:00.040+0000 7f1b1fec1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:00.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:00.044+0000 7f1b1fec1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:00.060 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:14:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:00Z" 2024-06-15T08:14:00.066 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:00.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:14:05.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:05.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:05.068 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:05.111 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:05.112 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:05.150 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:05.148+0000 7fe3bd42ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:05.151 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:05.152+0000 7fe3bd42ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:05.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:05.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:05.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:05.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:05.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:05.173 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:05.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:05.173 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:05Z" 2024-06-15T08:14:05.173 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:05.174 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:14:10.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:10.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:10.175 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:10.218 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:10.218 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:10.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:10.256+0000 7f9f6789aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:10.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:10.256+0000 7f9f6789aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:10Z" 2024-06-15T08:14:10.278 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:10.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:14:15.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:15.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:15.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:15.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:15.325 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:15.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:15.360+0000 7f23d758aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:15.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:15.360+0000 7f23d758aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:15.378 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:15.379 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:15.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:15.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:15.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:15.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:15.384 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:15Z" 2024-06-15T08:14:15.384 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:15.384 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:14:20.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:20.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:20.386 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:20.428 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:20.428 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:20.467 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:20.468+0000 7f7581294a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:20.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:20.468+0000 7f7581294a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:20.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:20.486 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:20.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:20.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:20.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:20Z" 2024-06-15T08:14:20.490 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:20.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:14:25.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:25.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:25.492 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:25.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:25.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:25.573 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:25.572+0000 7fa200a7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:25.574 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:25.572+0000 7fa200a7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:25.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:25.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:25.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:25.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:25.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:25Z" 2024-06-15T08:14:25.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:25.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:14:30.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:30.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:30.600 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:30.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:30.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:30.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:30.680+0000 7f5c146f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:30.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:30.680+0000 7f5c146f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:30.699 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:30.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:30.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.703 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:30.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:30.704 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:30.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:30.704 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:30Z" 2024-06-15T08:14:30.704 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:30.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:14:35.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:35.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:35.706 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:35.753 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:35.754 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:35.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:35.787+0000 7fc11d071a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:35.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:35.787+0000 7fc11d071a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:35.807 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:35.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:35.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:35.812 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:35.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:35.813 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:35Z" 2024-06-15T08:14:35.813 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:35.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:14:40.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:40.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:40.815 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:40.858 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:40.858 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:40.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:40.895+0000 7f7b23dada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:40.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:40.895+0000 7f7b23dada40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.913 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:40.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:40.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:40.917 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:40Z" 2024-06-15T08:14:40.918 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:40.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:14:45.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:45.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:45.920 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:45.969 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:45.970 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:46.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:46.015+0000 7f9de1d3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:46.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:46.015+0000 7f9de1d3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:46.038 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:46.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:46.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:46.043 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:46.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:46.043 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:46Z" 2024-06-15T08:14:46.043 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:46.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:14:51.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:51.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:51.045 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:51.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:51.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:51.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:51.127+0000 7efd2a0aba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:51.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:51.127+0000 7efd2a0aba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:51.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:51.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:51.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:51.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:51Z" 2024-06-15T08:14:51.150 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:51.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.920756_572.1 2024-06-15T08:14:56.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:56.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:14:56.153 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:14:56.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:14:56.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:14:56.239 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:56.235+0000 7f226569ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:56.240 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:14:56.239+0000 7f226569ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:14:56.256 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:56.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:56.258 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:14:56.259 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:14:56.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:14:56.261 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:14:56Z" 2024-06-15T08:14:56.261 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:14:56.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:01.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:01.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:01.263 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:01.309 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:01.309 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:01.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:01.343+0000 7fe56a05ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:01.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:01.347+0000 7fe56a05ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.364 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.365 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:01.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:01.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:01.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:01.369 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:01.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:01.369 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:01Z" 2024-06-15T08:15:01.369 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:01.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:06.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:06.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:06.371 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:06.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:06.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:06.452 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:06.451+0000 7f75d0fc2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:06.453 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:06.451+0000 7f75d0fc2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:06.469 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:15:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:06Z" 2024-06-15T08:15:06.474 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:06.475 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:11.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:11.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:11.476 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:11.519 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:11.519 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:11.556 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:11.555+0000 7f91364afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:11.556 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:11.555+0000 7f91364afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:11.572 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:11.573 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:11.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:11.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:11.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:11Z" 2024-06-15T08:15:11.577 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:11.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:16.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:16.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:16.580 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:16.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:16.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:16.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:16.659+0000 7f6115e83a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:16.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:16.659+0000 7f6115e83a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:16.679 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:16.679 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:16.679 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:16.679 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:16.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:16.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:16.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:16.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:16Z" 2024-06-15T08:15:16.684 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:16.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:21.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:21.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:21.686 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:21.730 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:21.730 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:21.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:21.767+0000 7f588f94da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:21.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:21.767+0000 7f588f94da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:21.786 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:21.786 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:21.786 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:21.786 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:21.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:21.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:21.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:21.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:21Z" 2024-06-15T08:15:21.791 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:21.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:15:26.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:26.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:26.793 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:26.837 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:26.837 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:26.874 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:26.870+0000 7f04cc70ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:26.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:26.874+0000 7f04cc70ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:26.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:26.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:26.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:26.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:26.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:26.897 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:26Z" 2024-06-15T08:15:26.897 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:26.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:15:31.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:31.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:31.899 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:31.957 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:31.957 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:32.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:32.010+0000 7fd58421ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:32.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:32.014+0000 7fd58421ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:32.031 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:32.031 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:32.031 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:32.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:32.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:32.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:32.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:32Z" 2024-06-15T08:15:32.036 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:32.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:15:37.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:37.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:37.038 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:37.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:37.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:37.120 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:37.118+0000 7ff47efe4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:37.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:37.118+0000 7ff47efe4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:37.140 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:37.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:37.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:37.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:37.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:37Z" 2024-06-15T08:15:37.145 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:37.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:15:42.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:42.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:42.150 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:42.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:42.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:42.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:42.246+0000 7f33a20bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:42.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:42.246+0000 7f33a20bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:42.264 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:42.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:42.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:42.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:42Z" 2024-06-15T08:15:42.269 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:42.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:15:47.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:47.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:47.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:47.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:47.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:47.356 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:47.354+0000 7f30b7923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:47.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:47.354+0000 7f30b7923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:47.373 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:47.373 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:47.374 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:47.375 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.374557_539.1", 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:47.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.374557Z", 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:47.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:47Z" 2024-06-15T08:15:47.378 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:47.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.374557_539.1 2024-06-15T08:15:52.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:52.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:52.381 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:52.422 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:52.422 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:52.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:52.458+0000 7fe0c72c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:52.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:52.458+0000 7fe0c72c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:52.477 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:52.477 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:52.477 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:52.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:52.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.734016_560.1", 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.734016Z", 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:52Z" 2024-06-15T08:15:52.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:52.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.734016_560.1 2024-06-15T08:15:57.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:57.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:15:57.485 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:15:57.527 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:15:57.528 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:15:57.566 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:57.562+0000 7ff2204d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:57.567 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:15:57.566+0000 7ff2204d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:15:57.584 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:15:57.584 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:57.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:57.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:15:57.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:15:57.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:15:57.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:15:57.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:15:57.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:15:57.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:15:57Z" 2024-06-15T08:15:57.590 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:15:57.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:16:02.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:02.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:02.593 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:02.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:02.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:02.673 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:02.670+0000 7f7b68dbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:02.674 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:02.670+0000 7f7b68dbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:02.690 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:02Z" 2024-06-15T08:16:02.695 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:02.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:07.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:07.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:07.697 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:07.746 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:07.746 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:07.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:07.786+0000 7f6e0bb9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:07.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:07.786+0000 7f6e0bb9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:07.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:07.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:07Z" 2024-06-15T08:16:07.812 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:07.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:12.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:12.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:12.814 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:12.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:12.858 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:12.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:12.894+0000 7fea8a9cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:12.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:12.898+0000 7fea8a9cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:12.912 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:12.912 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:12.912 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:12.912 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:12.912 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:12.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:12.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:12.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:12.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:12Z" 2024-06-15T08:16:12.917 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:12.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:17.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:17.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:17.920 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:17.964 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:17.964 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:18.003 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:18.002+0000 7f7807d33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:18.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:18.006+0000 7f7807d33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:18.020 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:18.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:18Z" 2024-06-15T08:16:18.025 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:18.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:16:23.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:23.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:23.031 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:23.074 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:23.075 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:23.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:23.113+0000 7fd15dd5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:23.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:23.113+0000 7fd15dd5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:23.131 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:23.131 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:23.131 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:23.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:23.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:23.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:23.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:23Z" 2024-06-15T08:16:23.136 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:23.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:28.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:28.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:28.139 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:28.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:28.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:28.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:28.221+0000 7feb320d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:28.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:28.225+0000 7feb320d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:28.241 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:28.241 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:28.241 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:28.241 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:28.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:28.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:28.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:28.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:28.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:28.247 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:28.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:28.247 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:28Z" 2024-06-15T08:16:28.247 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:28.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:33.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:33.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:33.250 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:33.292 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:33.292 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:33.331 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:33.333+0000 7f30522a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:33.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:33.333+0000 7f30522a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:33.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:33.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:33.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:33.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:33Z" 2024-06-15T08:16:33.355 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:33.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:16:38.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:38.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:38.357 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:38.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:38.400 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:38.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:38.437+0000 7f5d86b1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:38.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:38.441+0000 7f5d86b1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:38.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:38.458 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:38.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:38.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:38Z" 2024-06-15T08:16:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:38.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:43.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:43.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:43.464 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:43.508 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:43.508 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:43.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:43.545+0000 7fe2ca35fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:43.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:43.549+0000 7fe2ca35fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:43.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:43.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:43.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:43.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:43.569 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:43.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:43.569 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:43Z" 2024-06-15T08:16:43.569 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:43.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:16:48.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:48.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:48.571 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:48.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:48.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:48.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:48.665+0000 7f0d03c0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:48.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:48.665+0000 7f0d03c0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:48.687 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:48.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:48.687 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:48Z" 2024-06-15T08:16:48.687 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:48.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:16:53.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:53.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:53.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:53.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:53.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:53.773 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:53.773+0000 7f72dc809a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:53.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:53.773+0000 7f72dc809a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:53.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:53.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:53.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.380042_540.1", 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.380042Z", 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:53.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:53.795 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:53.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:53.796 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:53Z" 2024-06-15T08:16:53.796 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:53.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.380042_540.1 2024-06-15T08:16:58.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:58.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:16:58.798 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:16:58.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:16:58.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:16:58.878 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:58.877+0000 7f3decd1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:58.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:16:58.877+0000 7f3decd1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:58.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:58.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:16:58.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.900 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:16:58.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:16:58.901 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:16:58.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:16:58.901 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:16:58Z" 2024-06-15T08:16:58.901 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:16:58.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.742181_561.1 2024-06-15T08:17:03.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:03.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:03.903 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:03.946 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:03.946 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:03.985 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:03.985+0000 7f1b2ce39a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:03.986 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:03.985+0000 7f1b2ce39a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:04.003 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:04.003 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:04.003 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:04.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:04.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.765305_564.1", 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.765305Z", 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:04.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:04.007 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:03Z" 2024-06-15T08:17:04.008 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:04.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.765305_564.1 2024-06-15T08:17:09.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:09.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:09.010 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:09.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:09.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:09.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:09.092+0000 7feb6587aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:09.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:09.092+0000 7feb6587aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:09.111 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.765305_564.1", 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:09.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.765305Z", 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:09.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:09Z" 2024-06-15T08:17:09.114 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:09.115 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.765305_564.1 2024-06-15T08:17:14.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:14.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:14.116 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:14.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:14.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:14.242 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:14.240+0000 7f5307bcaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:14.243 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:14.244+0000 7f5307bcaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:14.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:14.263 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:14Z" 2024-06-15T08:17:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:14.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:17:19.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:19.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:19.266 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:19.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:19.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:19.349 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:19.348+0000 7fe168ca9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:19.350 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:19.348+0000 7fe168ca9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.367 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:19.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:19.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:19.371 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:19Z" 2024-06-15T08:17:19.372 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:19.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:17:24.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:24.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:24.375 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:24.417 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:24.417 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:24.455 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:24.456+0000 7ff3afc73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:24.456 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:24.456+0000 7ff3afc73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.366541_537.1", 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.366541Z", 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.477 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:24.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:24.478 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:24.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:24.478 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:24Z" 2024-06-15T08:17:24.478 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:24.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.366541_537.1 2024-06-15T08:17:29.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:29.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:29.480 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:29.524 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:29.524 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:29.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:29.560+0000 7f1b2971aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:29.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:29.564+0000 7f1b2971aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:29.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:29.585 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:29.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:29.585 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:29Z" 2024-06-15T08:17:29.585 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:29.586 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:17:34.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:34.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:17:34.588 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:17:34.631 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:34.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:34.669 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:34.668+0000 7f695b362a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:34.670 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:17:34.668+0000 7f695b362a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:34.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:34.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.403800_541.1", 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:34.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.403800Z", 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:17:34.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:17:34Z" 2024-06-15T08:17:34.691 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:34.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439152.829266_756.1 target=1_1718438536.403800_541.1 2024-06-15T08:17:39.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-06-15T08:17:39.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:17:39.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:17:39.695 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T08:17:39.737 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:39.737 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone2", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "test-zone2.rgw.meta:root", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "test-zone2.rgw.control", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2024-06-15T08:17:39.764 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "test-zone2.rgw.log", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "test-zone2.rgw.otp", 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2024-06-15T08:17:39.765 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "JU2SIBEYF8X99UFF", 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "8jaRkunVUlMedOfXv4IT94WQ6jmfTqYZ" 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2024-06-15T08:17:39.767 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:39.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:17:39.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:17:39.768 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-06-15T08:17:39.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:17:39.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:17:39.893 INFO:teuthology.orchestra.run.smithi131.stderr:Sending period to new master zone 7ac29e95-fe0d-461f-ae1e-cb1f85586a02 2024-06-15T08:17:39.939 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:17:39.939 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 4, 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": [], 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "period_map": { 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "zonegroups": [ 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "test-zonegroup", 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zonegroup", 2024-06-15T08:17:39.940 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": true, 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone2", 2024-06-15T08:17:39.941 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": false, 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2024-06-15T08:17:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone1", 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2024-06-15T08:17:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-zone3", 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi012.front.sepia.ceph.com:8001" 2024-06-15T08:17:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:17:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "short_zone_ids": [ 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 698550714 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:17:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3768332864 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 2892951031 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "period_config": { 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_quota": { 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "user_quota": { 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2024-06-15T08:17:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "user_ratelimit": { 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_ratelimit": { 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T08:17:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "anonymous_ratelimit": { 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_name": "test-realm", 2024-06-15T08:17:39.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:17:39.964 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:17:39.964 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2024-06-15T08:18:39.966 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-48 2024-06-15T08:18:40.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:40.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:40.045 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:18:40.104 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:18:40.105 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:18:40.237 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:18:40.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718439520.015783_810.1", 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:18:40.015783Z" 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:18:40.255 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:18:40.256 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:18:40.256 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:18:40.256 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:18:40.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:18:40.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:40.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:40.257 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:18:40.298 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:18:40.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:18:40.336 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:40.335+0000 7f312d1a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:40.337 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:40.335+0000 7f312d1a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:18:40.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:18:40.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:40.356 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:18:40.357 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:18:40.358 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:18:40.359 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:18:40.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:40.359 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:18:40Z" 2024-06-15T08:18:40.359 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:18:40.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:18:45.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:45.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:45.362 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:18:45.404 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:18:45.404 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:18:45.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:45.439+0000 7fd5e2a36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:45.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:45.443+0000 7fd5e2a36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:45.460 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:18:45.460 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:18:45.460 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:18:45.460 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:18:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:18:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:18:45Z" 2024-06-15T08:18:45.465 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:18:45.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:18:50.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:50.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:50.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:18:50.511 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:18:50.511 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:18:50.549 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:50.546+0000 7f0e3ad0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:50.550 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:50.546+0000 7f0e3ad0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:50.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:18:50.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:50.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:50.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:50.571 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:18:50Z" 2024-06-15T08:18:50.572 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:18:50.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:18:55.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:55.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:18:55.574 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:18:55.618 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:18:55.618 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:18:55.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:55.654+0000 7f448863ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:55.657 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:18:55.654+0000 7f448863ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:18:55.714 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:18:55.714 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:18:55.714 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:18:55.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:18:55.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:18:55.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.718 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:18:55.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:18:55.719 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:18:55.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:18:55.719 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:18:55Z" 2024-06-15T08:18:55.719 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:18:55.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:00.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:00.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:00.721 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:00.775 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:00.775 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:00.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:00.814+0000 7fecfb850a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:00.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:00.818+0000 7fecfb850a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:00.837 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:00.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:00.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:00.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:00.841 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:00Z" 2024-06-15T08:19:00.841 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:00.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:05.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:05.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:05.843 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:05.887 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:05.888 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:05.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:05.926+0000 7fac303e1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:05.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:05.926+0000 7fac303e1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:05.945 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:05.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:05.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:05.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:05.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:05.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:05.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:05.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:05.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:05.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:05Z" 2024-06-15T08:19:05.957 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:05.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:10.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:10.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:10.960 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:11.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:11.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:11.042 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:11.038+0000 7f9dc62b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:11.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:11.038+0000 7f9dc62b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:11.059 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:11.059 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:11.059 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:11.059 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:11.059 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:11.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:11.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:11.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:11.063 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:11Z" 2024-06-15T08:19:11.064 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:11.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:11.404 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T08:19:11.404+0000 7fe2cc05f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:19:16.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:16.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:16.066 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:16.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:16.109 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:16.149 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:16.146+0000 7f8440d73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:16.150 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:16.146+0000 7f8440d73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:16.167 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:16.167 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:16.167 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:16.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:16.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:16.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:16.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:16Z" 2024-06-15T08:19:16.172 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:16.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:21.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:21.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:21.175 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:21.217 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:21.218 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:21.255 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:21.254+0000 7f91c4dbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:21.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:21.254+0000 7f91c4dbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:21.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:21.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:21.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:21.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:21.278 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:21Z" 2024-06-15T08:19:21.278 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:21.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:26.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:26.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:26.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:26.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:26.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:26.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:26.362+0000 7fe2f386fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:26.367 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:26.362+0000 7fe2f386fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:26.416 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:26.416 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:26.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:26.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:26.419 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:26.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:26Z" 2024-06-15T08:19:26.421 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:26.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438537.989943_579.1 2024-06-15T08:19:31.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:31.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:31.423 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:31.466 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:31.466 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:31.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:31.502+0000 7f95bfc8da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:31.505 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:31.502+0000 7f95bfc8da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:31.521 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:31.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:31.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:31.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:31Z" 2024-06-15T08:19:31.526 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:31.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:36.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:36.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:36.529 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:36.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:36.572 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:36.610 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:36.610+0000 7fe1099d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:36.611 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:36.614+0000 7fe1099d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:36.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:36.630 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:36.631 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:36Z" 2024-06-15T08:19:36.632 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:36.633 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:41.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:41.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:41.635 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:41.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:41.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:41.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:41.718+0000 7efd71301a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:41.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:41.718+0000 7efd71301a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:41.734 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:41.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:41.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:41.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:41.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:41Z" 2024-06-15T08:19:41.739 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:41.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:46.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:46.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:46.741 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:46.784 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:46.785 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:46.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:46.825+0000 7fbc79b87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:46.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:46.825+0000 7fbc79b87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:46.841 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:46.841 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:46.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:46.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:46.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:46Z" 2024-06-15T08:19:46.847 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:46.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:19:51.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:51.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:51.849 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:51.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:51.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:51.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:51.929+0000 7fc4d0fcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:51.932 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:51.933+0000 7fc4d0fcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:51.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:51.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:51.948 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:51.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:51.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:51.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.952 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:51.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:51.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:51.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:51.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:51Z" 2024-06-15T08:19:51.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:51.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.955581_573.1 2024-06-15T08:19:56.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:56.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:19:56.955 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:19:56.999 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:19:56.999 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:19:57.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:57.037+0000 7f0b60adca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:57.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:19:57.041+0000 7f0b60adca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:19:57.056 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:19:57.056 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:19:57.056 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:19:57.056 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:57.057 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:19:57.058 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:19:57.059 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:19:57.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:19:57.061 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:19:57Z" 2024-06-15T08:19:57.061 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:19:57.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:20:02.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:02.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:02.062 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:02.144 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:02.145+0000 7ff2b98cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:02.145 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:02.145+0000 7ff2b98cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:02.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:02.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.166 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:02.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:02.167 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:02.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:02.167 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:02Z" 2024-06-15T08:20:02.167 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:02.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:20:07.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:07.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:07.169 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:07.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:07.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:07.251 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:07.249+0000 7f34bf6a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:07.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:07.253+0000 7f34bf6a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:07.268 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:07Z" 2024-06-15T08:20:07.273 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:07.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:20:12.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:12.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:12.274 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:12.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:12.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:12.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:12.357+0000 7f105f942a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:12.358 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:12.357+0000 7f105f942a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:12.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:12.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:12.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.379 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:12.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:12.380 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:12.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:12.380 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:12Z" 2024-06-15T08:20:12.380 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:12.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:20:17.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:17.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:17.382 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:17.423 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:17.423 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:17.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:17.461+0000 7f72c4279a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:17.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:17.461+0000 7f72c4279a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:17.477 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:17.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:17.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:17.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:17.481 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:17Z" 2024-06-15T08:20:17.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:17.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438539.076452_581.1 2024-06-15T08:20:22.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:22.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:22.484 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:22.528 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:22.528 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:22.566 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:22.565+0000 7f3523a44a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:22.567 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:22.569+0000 7f3523a44a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.584 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:22.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:22.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:22.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:22.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T08:20:22.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:22.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:22Z" 2024-06-15T08:20:22.591 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:22.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438539.179989_590.1 2024-06-15T08:20:27.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:27.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:27.594 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:27.638 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:27.638 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:27.676 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:27.677+0000 7fe3c8f0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:27.677 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:27.677+0000 7fe3c8f0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:27.693 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:27.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:27.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:27.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:27.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:27Z" 2024-06-15T08:20:27.699 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:27.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.728887_559.1 2024-06-15T08:20:32.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:32.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:32.701 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:32.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:32.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:32.797 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:32.797+0000 7f5cac7c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:32.798 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:32.797+0000 7f5cac7c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:32.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:32.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:32.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:32.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:32Z" 2024-06-15T08:20:32.823 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:32.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.728887_559.1 2024-06-15T08:20:37.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:37.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:37.825 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:37.868 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:37.868 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:37.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:37.904+0000 7f860dfc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:37.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:37.908+0000 7f860dfc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:37.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:37.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:37.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:37.928 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:37.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:37.929 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:37Z" 2024-06-15T08:20:37.929 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:37.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:20:42.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:42.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:42.931 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:42.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:42.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:43.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:43.012+0000 7f20b3650a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:43.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:43.012+0000 7f20b3650a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:43.030 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:43.030 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:43.030 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:43.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:43.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:43.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:43.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:43Z" 2024-06-15T08:20:43.035 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:43.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:20:48.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:48.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:48.038 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:48.081 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:48.081 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:48.120 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:48.120+0000 7fc185138a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:48.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:48.120+0000 7fc185138a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:48.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:48.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:48.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:48.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:48.143 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:48Z" 2024-06-15T08:20:48.143 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:48.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:20:53.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:53.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:53.146 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:53.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:53.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:53.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:53.228+0000 7fa406a3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:53.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:53.228+0000 7fa406a3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:53.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:53.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:53.249 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:53.250 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:53.251 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:53.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:53.252 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:53Z" 2024-06-15T08:20:53.252 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:53.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:20:58.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:58.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:20:58.254 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:20:58.297 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:20:58.297 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:20:58.351 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:58.352+0000 7fd1fea36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:58.352 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:20:58.352+0000 7fd1fea36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:20:58.368 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:20:58.368 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:20:58.368 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:20:58.368 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:58.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:58.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:58.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:20:58.372 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:20:58.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:20:58.374 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:20:58Z" 2024-06-15T08:20:58.374 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:20:58.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:03.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:03.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:03.377 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:03.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:03.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:03.459 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:03.456+0000 7f9bddd4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:03.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:03.460+0000 7f9bddd4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:03.477 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:03.477 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:03.477 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.478 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:03.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:03.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:03.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:03.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:03.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:03.485 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:03.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:03.486 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:03.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.486 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:03.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:03.487 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:03.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:03.487 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:03Z" 2024-06-15T08:21:03.487 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:03.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:08.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:08.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:08.490 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:08.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:08.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:08.572 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:08.572+0000 7f57d65a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:08.573 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:08.572+0000 7f57d65a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:08.590 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:08.590 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:08.590 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:08.590 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:08.590 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:08.591 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:08Z" 2024-06-15T08:21:08.595 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:08.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:13.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:13.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:13.598 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:13.640 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:13.640 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:13.679 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:13.680+0000 7fac92d07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:13.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:13.680+0000 7fac92d07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:13.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:13.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.701 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:13.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:13.702 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:13.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:13.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:13Z" 2024-06-15T08:21:13.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:13.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:18.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:18.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:18.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:18.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:18.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:18.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:18.784+0000 7ffb5f359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:18.787 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:18.784+0000 7ffb5f359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:18.804 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:18.804 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:18.804 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:18.804 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:18.804 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:18.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:18.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:18.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:18.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:18Z" 2024-06-15T08:21:18.809 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:18.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:23.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:23.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:23.812 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:23.855 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:23.855 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:23.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:23.896+0000 7f63678b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:23.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:23.896+0000 7f63678b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:23.914 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:23.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:23.914 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:23.914 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:23.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:23.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:23.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:23.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:23.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:23Z" 2024-06-15T08:21:23.919 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:23.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:21:28.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:28.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:28.922 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:28.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:28.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:29.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:29.004+0000 7f8f3b2cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:29.005 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:29.004+0000 7f8f3b2cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:29.023 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:29.023 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:29.023 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:29.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:29.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:29.026 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:29.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:29Z" 2024-06-15T08:21:29.028 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:29.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:21:34.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:34.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:34.031 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:34.073 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:34.073 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:34.112 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:34.111+0000 7f292b964a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:34.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:34.111+0000 7f292b964a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.130 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:34.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:21:34.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:34.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:34Z" 2024-06-15T08:21:34.134 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:34.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:21:39.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:39.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:39.136 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:39.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:39.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:39.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:39.223+0000 7fdee45eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:39.224 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:39.223+0000 7fdee45eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:39.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:39.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:39.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:39.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:39.246 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:39Z" 2024-06-15T08:21:39.247 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:39.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:21:44.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:44.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:44.248 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:44.291 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:44.291 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:44.330 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:44.327+0000 7f032202ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:44.331 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:44.327+0000 7f032202ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:44.348 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:44.348 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:44.348 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:44.348 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:44.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:44.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:44.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:44.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:44Z" 2024-06-15T08:21:44.353 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:44.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:49.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:49.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:49.356 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:49.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:49.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:49.437 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:49.435+0000 7ff646cc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:49.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:49.435+0000 7ff646cc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:49.455 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:49.455 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:49.455 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:49.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:49.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:49Z" 2024-06-15T08:21:49.461 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:49.462 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:21:54.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:54.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:54.463 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:54.508 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:54.508 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:54.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:54.543+0000 7f4a21dc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:54.548 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:54.547+0000 7f4a21dc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:54.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:54.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:54.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:54.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:54Z" 2024-06-15T08:21:54.569 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:54.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.699165_557.1 2024-06-15T08:21:59.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:59.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:21:59.572 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:21:59.615 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:21:59.615 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:21:59.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:59.655+0000 7f2110aa6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:59.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:21:59.655+0000 7f2110aa6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:21:59.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:21:59.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:21:59.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:21:59.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:21:59Z" 2024-06-15T08:21:59.678 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:21:59.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:04.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:04.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:04.680 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:04.725 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:04.725 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:04.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:04.763+0000 7f17204c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:04.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:04.763+0000 7f17204c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:04.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:04.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:04.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:04.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:04.786 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:04.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:04.786 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:04Z" 2024-06-15T08:22:04.786 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:04.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:22:09.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:09.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:09.789 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:09.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:09.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:09.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:09.883+0000 7fed67a49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:09.884 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:09.883+0000 7fed67a49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.901 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:09.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:09.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:09.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:09.906 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:09.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:09.906 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:09Z" 2024-06-15T08:22:09.906 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:09.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:22:14.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:14.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:14.909 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:14.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:14.952 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:14.991 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:14.991+0000 7f8e2f4fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:14.992 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:14.991+0000 7f8e2f4fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:15.009 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:15.010 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:15.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:15.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:15.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:14Z" 2024-06-15T08:22:15.014 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:15.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.707204_558.1 2024-06-15T08:22:20.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:20.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:20.017 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:20.061 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:20.061 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:20.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:20.099+0000 7fba9fa86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:20.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:20.099+0000 7fba9fa86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:20.117 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:20.118 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:20Z" 2024-06-15T08:22:20.123 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:20.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:25.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:25.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:25.126 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:25.171 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:25.171 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:25.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:25.206+0000 7f2a991f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:25.210 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:25.206+0000 7f2a991f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:25.226 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:25.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:25.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.631943_554.1", 2024-06-15T08:22:25.229 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.631943Z", 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:25.230 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:25Z" 2024-06-15T08:22:25.231 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:25.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.631943_554.1 2024-06-15T08:22:30.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:30.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:30.234 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:30.279 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:30.279 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:30.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:30.318+0000 7f367a781a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:30.321 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:30.318+0000 7f367a781a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:30.338 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:30.338 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:30.338 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:30.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:30Z" 2024-06-15T08:22:30.344 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:30.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:35.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:35.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:35.346 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:35.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:35.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:35.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:35.430+0000 7f028a60fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:35.432 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:35.430+0000 7f028a60fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:35.449 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:35.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:35.454 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:35Z" 2024-06-15T08:22:35.455 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:35.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:40.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:40.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:40.457 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:40.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:40.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:40.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:40.538+0000 7f4f6d424a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:40.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:40.538+0000 7f4f6d424a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:40Z" 2024-06-15T08:22:40.563 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:40.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:45.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:45.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:45.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:45.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:45.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:45.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:45.646+0000 7ffa61ac8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:45.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:45.646+0000 7ffa61ac8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:45.666 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:45.666 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:45.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:45.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:45.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:45.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:45Z" 2024-06-15T08:22:45.671 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:45.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:50.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:50.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:50.674 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:50.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:50.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:50.757 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:50.754+0000 7f27c87dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:50.757 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:50.754+0000 7f27c87dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:50.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:50.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:50.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:50.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:50Z" 2024-06-15T08:22:50.778 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:50.779 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:22:55.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:55.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:22:55.780 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:22:55.837 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:22:55.837 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:22:55.880 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:55.878+0000 7f2345a48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:55.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:22:55.878+0000 7f2345a48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:22:55.898 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:22:55.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:55.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:55.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:22:55.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:22:55.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:22:55.904 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:22:55Z" 2024-06-15T08:22:55.904 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:22:55.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:00.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:00.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:00.906 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:00.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:00.949 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:00.988 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:00.986+0000 7ff8044c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:00.990 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:00.986+0000 7ff8044c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:01.007 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:01.007 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:01.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:01.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:01.011 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:01.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:01.012 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:00Z" 2024-06-15T08:23:01.012 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:01.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:23:06.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:06.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:06.014 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:06.055 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:06.056 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:06.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:06.090+0000 7f1d2b900a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:06.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:06.090+0000 7f1d2b900a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:06.111 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:06.111 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:06.111 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:06.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:06.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:06.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:06Z" 2024-06-15T08:23:06.116 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:06.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.895427_569.1 2024-06-15T08:23:11.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:11.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:11.118 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:11.161 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:11.161 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:11.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:11.198+0000 7f015be2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:11.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:11.202+0000 7f015be2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.217 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:11.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:11.220 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:11.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:11.222 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:11Z" 2024-06-15T08:23:11.222 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:11.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:23:16.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:16.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:16.224 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:16.269 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:16.269 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:16.308 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:16.310+0000 7f0249e52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:16.309 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:16.310+0000 7f0249e52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:16.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:16.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:16.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:16.332 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:16Z" 2024-06-15T08:23:16.333 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:16.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:23:21.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:21.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:21.335 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:21.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:21.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:21.417+0000 7f937e423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:21.419 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:21.421+0000 7f937e423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:21.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:21.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:21.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:21.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:21.441 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:21.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:21.442 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:21Z" 2024-06-15T08:23:21.442 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:21.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:26.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:26.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:26.444 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:26.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:26.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:26.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:26.521+0000 7f36b3d0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:26.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:26.525+0000 7f36b3d0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:26.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:26.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:26.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:26Z" 2024-06-15T08:23:26.546 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:26.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.728887_559.1 2024-06-15T08:23:31.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:31.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:31.548 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:31.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:31.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:31.641 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:31.641+0000 7f273a658a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:31.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:31.641+0000 7f273a658a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:31.659 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:31.659 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:31.659 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:31.659 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:31.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:31.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.696719_556.1", 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:31.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.696719Z", 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:31.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:31Z" 2024-06-15T08:23:31.664 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:31.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.696719_556.1 2024-06-15T08:23:36.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:36.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:36.667 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:36.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:36.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:36.749 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:36.749+0000 7fe01f45aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:36.749 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:36.749+0000 7fe01f45aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:36.766 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:36.767 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:36.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:36.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:36.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:36.771 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:36Z" 2024-06-15T08:23:36.772 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:36.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:41.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:41.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:41.774 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:41.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:41.816 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:41.854 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:41.853+0000 7fc097bfaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:41.855 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:41.857+0000 7fc097bfaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:41.873 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:41.874 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:41.875 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:41.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:41Z" 2024-06-15T08:23:41.877 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:41.878 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:46.879 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-06-15T08:23:46.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:46.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:46.880 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:46.942 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:47.038 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718439520.015783_810.1", 2024-06-15T08:23:47.057 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:18:40.015783Z" 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:23:47.058 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:23:47.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:23:47.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:47.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:47.059 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:47.103 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:47.104 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:47.145 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:47.145+0000 7f6b91899a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:47.146 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:47.145+0000 7f6b91899a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:47.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:47.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:47.171 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:47.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:47.172 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:47Z" 2024-06-15T08:23:47.172 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:47.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:52.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:52.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:52.174 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:52.219 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:52.220 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:52.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:52.257+0000 7f6ecb8ffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:52.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:52.261+0000 7f6ecb8ffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:52.276 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:52.276 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:52.276 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:52.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:52.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:52.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:52.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:52.281 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:52.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:52.282 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:52Z" 2024-06-15T08:23:52.282 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:52.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:23:57.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:57.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:23:57.284 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:23:57.334 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:23:57.334 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:23:57.383 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:57.385+0000 7f7ac36a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:57.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:23:57.385+0000 7f7ac36a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:57.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:57.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:23:57.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:23:57.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:23:57.406 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:23:57.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:23:57.406 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:23:57Z" 2024-06-15T08:23:57.406 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:23:57.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:02.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:02.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:02.409 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:02.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:02.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:02.493 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:02.493+0000 7f24261a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:02.494 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:02.493+0000 7f24261a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:02.512 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:02.512 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:02.512 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:02.512 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:02.512 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:02.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:02.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:02.515 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:02.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:02.517 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:02Z" 2024-06-15T08:24:02.517 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:02.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:24:07.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:07.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:07.519 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:07.563 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:07.563 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:07.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:07.625+0000 7f6f212f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:07.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:07.625+0000 7f6f212f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:07.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:07.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:07.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:07.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:07Z" 2024-06-15T08:24:07.670 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:07.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:24:12.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:12.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:12.673 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:12.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:12.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:12.771 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:12.769+0000 7f478ad45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:12.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:12.773+0000 7f478ad45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:12.790 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:12.790 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:12.790 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:12.790 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:12.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:12.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:12.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:12.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:12Z" 2024-06-15T08:24:12.795 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:12.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:17.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:17.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:17.797 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:17.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:17.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:17.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:17.880+0000 7f8d2c44aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:17.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:17.884+0000 7f8d2c44aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:17.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:17.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:17.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:17.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:17.906 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:17Z" 2024-06-15T08:24:17.907 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:17.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:24:22.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:22.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:22.909 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:22.952 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:22.952 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:22.996 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:22.996+0000 7fe6c1f08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:22.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:22.996+0000 7fe6c1f08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:23.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:23.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:23Z" 2024-06-15T08:24:23.020 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:23.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:28.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:28.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:28.023 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:28.067 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:28.067 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:28.109 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:28.108+0000 7f6470f89a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:28.110 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:28.108+0000 7f6470f89a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:28.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:28.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:28.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:28.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:28.134 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:28.135 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:28Z" 2024-06-15T08:24:28.136 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:28.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.728887_559.1 2024-06-15T08:24:31.436 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T08:24:31.421+0000 7fe2cc05f640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:24:33.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:33.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:33.138 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:33.183 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:33.183 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:33.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:33.224+0000 7fec45449a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:33.227 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:33.228+0000 7fec45449a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:33.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:33.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:33.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:33.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:33Z" 2024-06-15T08:24:33.263 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:33.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:38.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:38.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:38.266 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:38.308 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:38.309 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:38.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:38.368+0000 7f804956ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:38.369 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:38.368+0000 7f804956ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.392 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:38.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:38.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:38Z" 2024-06-15T08:24:38.397 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:38.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:43.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:43.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:43.399 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:43.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:43.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:43.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:43.520+0000 7fc8925aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:43.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:43.520+0000 7fc8925aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:43.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:43.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:43.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:43.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:43.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:43.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:43Z" 2024-06-15T08:24:43.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:43.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:24:48.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:48.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:48.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:48.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:48.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:48.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:48.636+0000 7f72df6dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:48.638 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:48.636+0000 7f72df6dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.658 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:48.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.621936_552.1", 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.621936Z", 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:48.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:48.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:48.663 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:48.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:48.663 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:48Z" 2024-06-15T08:24:48.663 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:48.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.621936_552.1 2024-06-15T08:24:53.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:53.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:53.665 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:53.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:53.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:53.751 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:53.752+0000 7ff8f8f99a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:53.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:53.752+0000 7ff8f8f99a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:53.770 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:53.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:53Z" 2024-06-15T08:24:53.775 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:53.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:24:58.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:58.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:24:58.778 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:24:58.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:24:58.821 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:24:58.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:58.860+0000 7f2f70c45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:24:58.864+0000 7f2f70c45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:24:58.902 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:58.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:58.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:24:58.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:24:58.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:24:58.907 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:24:58Z" 2024-06-15T08:24:58.908 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:24:58.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:25:03.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:03.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:03.910 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:03.954 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:03.954 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:04.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:04.016+0000 7f0d96423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:04.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:04.016+0000 7f0d96423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:04.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:04.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:04.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:04.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:04Z" 2024-06-15T08:25:04.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:04.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:25:09.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:09.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:09.047 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:09.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:09.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:09.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:09.136+0000 7fa1de028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:09.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:09.136+0000 7fa1de028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:09.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:09.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:09Z" 2024-06-15T08:25:09.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:09.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:25:14.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:14.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:14.166 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:14.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:14.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:14.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:14.255+0000 7f4f6b9bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:14.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:14.255+0000 7f4f6b9bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:14.275 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:14.275 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:14.275 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:14.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:14.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:14.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:14.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:14Z" 2024-06-15T08:25:14.280 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:14.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:25:19.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:19.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:19.282 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:19.327 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:19.327 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:19.370 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:19.367+0000 7f3d066c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:19.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:19.371+0000 7f3d066c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:19.389 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:19.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:19.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:19.394 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:19Z" 2024-06-15T08:25:19.395 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:19.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:25:24.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:24.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:24.398 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:24.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:24.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:24.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:24.487+0000 7f8b467dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:24.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:24.487+0000 7f8b467dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:24.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:24.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:24.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:24.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:24.525 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:24.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:24.525 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:24Z" 2024-06-15T08:25:24.525 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:24.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:25:29.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:29.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:29.527 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:29.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:29.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:29.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:29.611+0000 7f39f1ea9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:29.614 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:29.611+0000 7f39f1ea9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.634 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:29.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:29.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:29.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:29Z" 2024-06-15T08:25:29.639 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:29.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:25:34.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:34.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:34.642 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:34.691 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:34.692 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:34.733 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:34.731+0000 7fa6e7316a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:34.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:34.731+0000 7fa6e7316a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:34.751 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:34.752 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.753 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:34.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:34.755 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:34.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:34Z" 2024-06-15T08:25:34.757 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:34.758 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:25:39.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:39.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:39.760 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:39.803 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:39.803 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:39.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:39.859+0000 7f5ba2473a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:39.861 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:39.859+0000 7f5ba2473a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:39.885 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:39.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:39.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:39.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:39.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:39.890 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:39.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:39.891 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:39Z" 2024-06-15T08:25:39.891 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:39.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:25:44.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:44.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:44.893 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:44.936 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:44.936 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:44.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:44.975+0000 7fae4c191a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:44.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:44.975+0000 7fae4c191a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:44.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:44.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:44.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:44.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:45.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:45.001 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:44Z" 2024-06-15T08:25:45.002 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:45.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:25:50.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:50.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:50.004 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:50.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:50.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:50.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:50.111+0000 7f40f32b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:50.116 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:50.115+0000 7f40f32b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:50.138 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:50.138 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:50.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:50.140 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:50.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:50.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:50Z" 2024-06-15T08:25:50.143 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:50.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:25:55.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:55.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:25:55.146 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:25:55.192 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:25:55.192 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:25:55.239 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:55.239+0000 7f311396ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:55.240 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:25:55.239+0000 7f311396ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:25:55.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:25:55.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:25:55.259 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:55.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:25:55.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:55.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:25:55.263 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:25:55Z" 2024-06-15T08:25:55.264 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:25:55.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:26:00.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:00.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:00.266 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:00.313 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:00.314 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:00.363 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:00.363+0000 7fce34dc1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:00.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:00.363+0000 7fce34dc1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:00.381 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:00.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:00Z" 2024-06-15T08:26:00.386 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:00.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:05.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:05.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:05.389 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:05.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:05.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:05.472 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:05.471+0000 7f4f6b8f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:05.473 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:05.471+0000 7f4f6b8f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:05.491 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:05.491 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:05.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:05.494 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:05.495 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:05.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.624727_553.1", 2024-06-15T08:26:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.624727Z", 2024-06-15T08:26:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:05Z" 2024-06-15T08:26:05.503 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:05.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.624727_553.1 2024-06-15T08:26:10.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:10.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:10.506 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:10.550 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:10.550 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:10.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:10.590+0000 7f505a095a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:10.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:10.590+0000 7f505a095a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:10.609 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:10.609 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:10.610 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:10.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:10.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:10.613 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:10Z" 2024-06-15T08:26:10.614 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:10.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:15.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:15.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:15.616 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:15.661 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:15.661 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:15.700 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:15.698+0000 7f21283aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:15.701 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:15.698+0000 7f21283aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:15.719 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:15.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:15.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:15.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:15.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:15Z" 2024-06-15T08:26:15.724 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:15.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.742181_561.1 2024-06-15T08:26:20.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:20.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:20.727 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:20.771 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:20.771 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:20.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:20.822+0000 7f374ba84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:20.826 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:20.822+0000 7f374ba84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:20.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:20.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:20.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:20.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:20.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:20Z" 2024-06-15T08:26:20.850 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:20.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:25.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:25.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:25.852 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:25.937 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:25.934+0000 7f961c8e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:25.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:25.934+0000 7f961c8e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:25.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:25.958 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:25.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:25.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:25.961 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:25Z" 2024-06-15T08:26:25.961 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:25.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:30.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:30.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:30.963 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:31.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:31.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:31.047 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:31.046+0000 7fcd885c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:31.048 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:31.046+0000 7fcd885c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:31.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:31.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:31.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.073 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:31.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:31.074 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:31.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:31.074 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:31Z" 2024-06-15T08:26:31.074 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:31.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:36.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:36.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:36.077 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:36.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:36.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:36.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:36.158+0000 7f865b356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:36.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:36.158+0000 7f865b356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:36.180 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:36.180 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:36.180 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:36.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:26:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:36Z" 2024-06-15T08:26:36.188 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:36.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.699165_557.1 2024-06-15T08:26:41.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:41.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:41.191 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:41.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:41.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:41.279 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:41.278+0000 7f3e5020fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:41.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:41.278+0000 7f3e5020fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:41.306 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:41.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:41.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.310 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:41.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:41.311 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:41.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:41.311 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:41Z" 2024-06-15T08:26:41.311 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:41.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:46.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:46.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:46.313 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:46.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:46.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:46.411 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:46.410+0000 7fccf7bd8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:46.411 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:46.410+0000 7fccf7bd8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:46.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:46.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:46.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:46.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:46Z" 2024-06-15T08:26:46.437 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:46.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:51.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:51.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:51.439 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:51.480 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:51.480 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:51.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:51.518+0000 7f9216c29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:51.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:51.518+0000 7f9216c29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:51.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:51.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:51.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:51.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:51.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:51.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:51Z" 2024-06-15T08:26:51.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:51.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:26:56.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:56.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:26:56.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:26:56.591 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:26:56.591 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:26:56.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:56.630+0000 7fcb180e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:56.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:26:56.634+0000 7fcb180e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:56.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:26:56.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.658 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:26:56.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:26:56.659 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:26:56.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:26:56.659 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:26:56Z" 2024-06-15T08:26:56.659 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:26:56.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:27:01.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:01.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:01.662 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:01.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:01.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:01.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:01.762+0000 7ff193959a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:01.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:01.762+0000 7ff193959a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:01.785 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:01.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:01.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:01.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:01.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:01.790 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:01.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:01.791 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:01Z" 2024-06-15T08:27:01.791 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:01.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:27:06.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:06.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:06.794 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:06.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:06.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:06.912 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:06.913+0000 7f6639c62a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:06.913 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:06.913+0000 7f6639c62a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:06.930 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:06.930 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:06.930 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:06.930 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:06.930 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:06.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:06.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:06.933 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:06.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:06.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:06.936 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:06Z" 2024-06-15T08:27:06.936 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:06.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:27:11.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:11.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:11.938 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:11.981 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:11.981 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:12.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:12.045+0000 7fdac58a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:12.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:12.045+0000 7fdac58a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:12.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:12.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:12.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:12.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:12Z" 2024-06-15T08:27:12.075 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:12.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:27:17.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:17.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:17.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:17.122 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:17.122 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:17.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:17.161+0000 7f9503343a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:17.163 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:17.165+0000 7f9503343a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:17.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:17.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:17.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:17.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:17Z" 2024-06-15T08:27:17.185 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:17.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:27:22.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:22.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:22.188 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:22.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:22.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:22.289 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:22.289+0000 7f7871dbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:22.290 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:22.289+0000 7f7871dbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:22.310 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:22.310 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:22.310 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:22.310 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:22.310 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:22.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:22.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:27:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:22.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.315 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:22.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:22.316 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:22.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:22.316 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:22Z" 2024-06-15T08:27:22.316 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:22.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:27:27.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:27.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:27.319 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:27.362 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:27.362 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:27.411 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:27.413+0000 7f54f34f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:27.412 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:27.413+0000 7f54f34f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.431 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:27.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:27.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:27.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:27Z" 2024-06-15T08:27:27.436 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:27.437 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:27:32.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:32.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:32.438 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:32.481 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:32.481 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:32.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:32.521+0000 7f7ff24c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:32.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:32.525+0000 7f7ff24c2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:32.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:32.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:32.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:27:32.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:32Z" 2024-06-15T08:27:32.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:32.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.728887_559.1 2024-06-15T08:27:37.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:37.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:37.548 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:37.591 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:37.591 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:37.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:37.633+0000 7faa2f23ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:37.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:37.633+0000 7faa2f23ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:37.649 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:37.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:37.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:37.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:37Z" 2024-06-15T08:27:37.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:37.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438539.076452_581.1 2024-06-15T08:27:42.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:42.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:42.658 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:42.701 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:42.701 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:42.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:42.761+0000 7ffb07d1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:42.763 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:42.765+0000 7ffb07d1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:42.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:42.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:42.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:42.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:42Z" 2024-06-15T08:27:42.794 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:42.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.699165_557.1 2024-06-15T08:27:47.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:47.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:47.796 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:47.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:47.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:47.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:47.881+0000 7f41994c6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:47.880 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:47.881+0000 7f41994c6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:47.899 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:47.899 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:47.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:47.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:47.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:47.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:47Z" 2024-06-15T08:27:47.904 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:47.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.699165_557.1 2024-06-15T08:27:52.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:52.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:52.906 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:52.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:52.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:53.006 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:53.005+0000 7f3c8d670a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:53.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:53.009+0000 7f3c8d670a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:53.029 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:53.030 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:53.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:53.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:53.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:53Z" 2024-06-15T08:27:53.034 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:53.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:27:58.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:58.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:27:58.037 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:27:58.079 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:27:58.079 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:27:58.122 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:58.121+0000 7f234a50aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:58.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:27:58.125+0000 7f234a50aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:27:58.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:27:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:27:58.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:27:58Z" 2024-06-15T08:27:58.153 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:27:58.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:03.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:03.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:03.156 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:03.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:03.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:03.268 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:03.268+0000 7fe8bc1b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:03.269 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:03.268+0000 7fe8bc1b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.287 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:03.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:03.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:03.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:03Z" 2024-06-15T08:28:03.291 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:03.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:08.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:08.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:08.293 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:08.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:08.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:08.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:08.388+0000 7fba92081a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:08.391 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:08.392+0000 7fba92081a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:08.412 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:08.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:08.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:08.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:08.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:08Z" 2024-06-15T08:28:08.417 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:08.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:13.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:13.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:13.420 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:13.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:13.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:13.511 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:13.512+0000 7f9fd3b14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:13.512 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:13.512+0000 7f9fd3b14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:13.531 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:13Z" 2024-06-15T08:28:13.536 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:13.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:18.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:18.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:18.538 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:18.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:18.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:18.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:18.620+0000 7fc656eeba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:18.621 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:18.620+0000 7fc656eeba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:18.639 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:18.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:18.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:18.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:18.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:18Z" 2024-06-15T08:28:18.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:18.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.690655_555.1 2024-06-15T08:28:23.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:23.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:23.646 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:23.691 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:23.691 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:23.735 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:23.736+0000 7f55ff097a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:23.735 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:23.736+0000 7f55ff097a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:23.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:23.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.758 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:23.759 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.760 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:23.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:23.761 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:23.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:23.761 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:23Z" 2024-06-15T08:28:23.761 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:23.762 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:28.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:28.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:28.763 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:28.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:28.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:28.898 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:28.896+0000 7f2c5ffd3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:28.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:28.900+0000 7f2c5ffd3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:28.916 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:28.916 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:28.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:28.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:28.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:28.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:28Z" 2024-06-15T08:28:28.921 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:28.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:33.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:33.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:33.924 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:33.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:33.969 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:34.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:34.012+0000 7f09aaf94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:34.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:34.016+0000 7f09aaf94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:34.033 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:34.033 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:34.033 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:34.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:34.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:34.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:34.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:34Z" 2024-06-15T08:28:34.038 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:34.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:39.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:39.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:39.040 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:39.084 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:39.084 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:39.139 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:39.140+0000 7f171af6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:39.140 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:39.140+0000 7f171af6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:39.163 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:39.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:39.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:39.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:39.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:39Z" 2024-06-15T08:28:39.168 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:39.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:44.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:44.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:44.171 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:44.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:44.256+0000 7effbdb92a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:44.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:44.256+0000 7effbdb92a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:44.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:44.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:44.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.572026_547.1", 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.572026Z", 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:44.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:44.298 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:44Z" 2024-06-15T08:28:44.299 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:44.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.572026_547.1 2024-06-15T08:28:49.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:49.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:49.301 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:49.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:49.345 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:49.397 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:49.395+0000 7fea50e57a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:49.398 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:28:49.395+0000 7fea50e57a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:28:49.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:49.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:28:49.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:28:49.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:28:49Z" 2024-06-15T08:28:49.428 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:28:49.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718439520.015783_810.1 target=1_1718438536.615034_551.1 2024-06-15T08:28:54.431 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2024-06-15T08:28:54.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.432 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:54.507 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:54.595 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "183eec1b-0866-4613-9a5b-8d29e0fcfe81", 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "remove_zg", 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "remove_zg", 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": false, 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [], 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "", 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [], 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [] 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.597 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:28:54.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.599 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:54.628 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:54.628 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:54.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:28:54.663+0000 7f114d038a40 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-06-15T08:28:54.671 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:28:54.671 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "70ec21a2-96c8-446c-90c9-bd15269d8c44", 2024-06-15T08:28:54.671 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "remove", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "domain_root": "remove.rgw.meta:root", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "control_pool": "remove.rgw.control", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "gc_pool": "remove.rgw.log:gc", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "lc_pool": "remove.rgw.log:lc", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "log_pool": "remove.rgw.log", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-06-15T08:28:54.672 INFO:teuthology.orchestra.run.smithi012.stdout: "otp_pool": "remove.rgw.otp", 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "system_key": { 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "", 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "" 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_pools": [ 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "default-placement", 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "val": { 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "index_pool": "remove.rgw.buckets.index", 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": { 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD": { 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: "data_pool": "remove.rgw.buckets.data" 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: "index_type": 0, 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: "inline_data": true 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout: "notif_pool": "remove.rgw.log:notif" 2024-06-15T08:28:54.674 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:28:54.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.675 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:54.750 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:54.751 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:54.836 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:28:54.836 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:54.836 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 5, 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": [], 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "period_map": { 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "zonegroups": [ 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "183eec1b-0866-4613-9a5b-8d29e0fcfe81", 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "remove_zg", 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "remove_zg", 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": false, 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [], 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T08:28:54.837 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "70ec21a2-96c8-446c-90c9-bd15269d8c44", 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [ 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "70ec21a2-96c8-446c-90c9-bd15269d8c44", 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "remove", 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [], 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": false, 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:54.838 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [ 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD" 2024-06-15T08:28:54.839 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.840 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "test-zonegroup", 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zonegroup", 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "", 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": true, 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [ 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:28:54.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone2", 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": false, 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:54.842 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:54.843 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.844 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.856 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.856 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.856 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:28:54.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone3", 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8001" 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:54.857 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [ 2024-06-15T08:28:54.858 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD" 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.859 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "short_zone_ids": [ 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "70ec21a2-96c8-446c-90c9-bd15269d8c44", 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 3341337480 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 698550714 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 3768332864 2024-06-15T08:28:54.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 2892951031 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "period_config": { 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T08:28:54.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "user_ratelimit": { 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:54.862 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_ratelimit": { 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "anonymous_ratelimit": { 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:54.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_name": "test-realm", 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:28:54.864 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:28:54.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.866 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:54.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:54.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:28:54.967+0000 7f9112042a40 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-06-15T08:28:54.972 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:28:54.971+0000 7f9112042a40 0 Removed zone from zonegroup remove_zg 2024-06-15T08:28:54.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:54.983 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:55.062 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:55.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-06-15T08:28:55.111+0000 7f3ff0c12a40 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-06-15T08:28:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-15T08:28:55.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:55.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:55.123 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:55.200 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:55.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:55.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-06-15T08:28:55.256 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-06-15T08:28:55.367 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:55.368 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:55.603 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 6, 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "predecessor_uuid": "6efc1a3f-742f-4f4f-9712-58131a5a0680", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_status": [], 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "period_map": { 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "zonegroups": [ 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "test-zonegroup", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zonegroup", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "api_name": "", 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "is_master": true, 2024-06-15T08:28:55.604 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames": [], 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "hostnames_s3website": [], 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "zones": [ 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone2", 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.605 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": false, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.606 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone1", 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8000" 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:55.607 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "id": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test-zone3", 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "endpoints": [ 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "http://smithi012.front.sepia.ceph.com:8001" 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "log_meta": false, 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "log_data": true, 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_index_max_shards": 11, 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "read_only": false, 2024-06-15T08:28:55.608 INFO:teuthology.orchestra.run.smithi012.stdout: "tier_type": "", 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from_all": true, 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_from": [], 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_zone": "", 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "supported_features": [ 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "compress-encrypted", 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_targets": [ 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default-placement", 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-06-15T08:28:55.609 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_classes": [ 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "STANDARD" 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "default-placement", 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "sync_policy": { 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "groups": [] 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled_features": [ 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: "resharding" 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.610 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: "short_zone_ids": [ 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "75a221df-8373-409f-a0c0-681fe8b4f342", 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 698550714 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:55.611 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 3768332864 2024-06-15T08:28:55.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.636 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:55.636 INFO:teuthology.orchestra.run.smithi012.stdout: "key": "ec1d75a7-d3e8-402a-8fe3-454f79b9554f", 2024-06-15T08:28:55.636 INFO:teuthology.orchestra.run.smithi012.stdout: "val": 2892951031 2024-06-15T08:28:55.636 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zonegroup": "test-zonegroup", 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "master_zone": "7ac29e95-fe0d-461f-ae1e-cb1f85586a02", 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "period_config": { 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.637 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "user_ratelimit": { 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:55.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_ratelimit": { 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "anonymous_ratelimit": { 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_ops": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_ops": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_read_bytes": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max_write_bytes": 0, 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false 2024-06-15T08:28:55.639 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:55.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:55.640 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_id": "c43304c1-0b90-4ac6-a459-45e84b6191f4", 2024-06-15T08:28:55.640 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_name": "test-realm", 2024-06-15T08:28:55.640 INFO:teuthology.orchestra.run.smithi012.stdout: "realm_epoch": 2 2024-06-15T08:28:55.640 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-06-15T08:28:55.641 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-06-15T08:28:55.641 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-49 2024-06-15T08:28:56.788 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-50 2024-06-15T08:28:56.904 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-51 2024-06-15T08:28:56.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-06-15T08:28:56.994 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-52 2024-06-15T08:28:57.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-53 2024-06-15T08:28:57.156 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-54 2024-06-15T08:28:59.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-06-15T08:28:59.403 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-55 2024-06-15T08:28:59.475 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-56 2024-06-15T08:28:59.560 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-57 2024-06-15T08:28:59.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:59.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:59.621 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:59.780 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:28:59.870 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:28:59.894 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718440139.598802_945.1", 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:28:59.598802Z" 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:28:59.895 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:28:59.896 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:28:59.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:59.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:28:59.897 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:28:59.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:28:59.940 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:00.042 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:00.043+0000 7efcaf387a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:00.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:00.043+0000 7efcaf387a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:00.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:00.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:00.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:00Z" 2024-06-15T08:29:00.067 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:00.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:29:05.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:05.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:05.070 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:05.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:05.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:05.183 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:05.183+0000 7f6da935da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:05.184 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:05.183+0000 7f6da935da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:05.201 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:05.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:05.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:05.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:05.205 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:05Z" 2024-06-15T08:29:05.206 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:05.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:29:10.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:10.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:10.208 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:10.252 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:10.252 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:10.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:10.295+0000 7f0e6d1c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:10.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:10.299+0000 7f0e6d1c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.328 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:10.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:10.329 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:10.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:10.329 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:10Z" 2024-06-15T08:29:10.329 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:10.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:29:15.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:15.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:15.332 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:15.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:15.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:15.429 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:15.427+0000 7f449127da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:15.430 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:15.427+0000 7f449127da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:15.450 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:15.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:15.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:15.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:15Z" 2024-06-15T08:29:15.455 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:15.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:29:20.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:20.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:20.457 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:20.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:20.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:20.544 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:20.543+0000 7f7e44c38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:20.545 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:20.543+0000 7f7e44c38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:29:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:20Z" 2024-06-15T08:29:20.568 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:20.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:29:25.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:25.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:25.570 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:25.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:25.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:25.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:25.663+0000 7f3fd7039a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:25.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:25.667+0000 7f3fd7039a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:25.688 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:25.693 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:25.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:25.694 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:25Z" 2024-06-15T08:29:25.694 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:25.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:29:30.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:30.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:30.696 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:30.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:30.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:30.801 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:30.799+0000 7fd551844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:30.802 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:30.803+0000 7fd551844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:30.821 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:30.821 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:30.821 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:30.821 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:30.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:30.823 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:30.824 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:30.825 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:30Z" 2024-06-15T08:29:30.826 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:30.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:29:35.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:35.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:35.829 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:35.872 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:35.872 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:35.934 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:35.930+0000 7fd96eb54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:35.935 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:35.934+0000 7fd96eb54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:35.960 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:35Z" 2024-06-15T08:29:35.964 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:35.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:29:40.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:40.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:40.967 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:41.016 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:41.017 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:41.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:41.062+0000 7f304702fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:41.065 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:41.062+0000 7f304702fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:41.086 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:41.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:41.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T08:29:41.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:41.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:41Z" 2024-06-15T08:29:41.091 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:41.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.759965_563.1 2024-06-15T08:29:46.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:46.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:46.093 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:46.136 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:46.136 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:46.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:46.182+0000 7fe22cf6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:46.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:46.186+0000 7fe22cf6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:46.207 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:46.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:46.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:46.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:46.212 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:46.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:46.212 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:46Z" 2024-06-15T08:29:46.212 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:46.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.895427_569.1 2024-06-15T08:29:51.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:51.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:51.214 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:51.258 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:51.258 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:51.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:51.326+0000 7f071808aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:51.329 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:51.326+0000 7f071808aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:51.350 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:51.350 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:51.350 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:51.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:51.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:51.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:51.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:51Z" 2024-06-15T08:29:51.355 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:51.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:29:56.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:56.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:29:56.358 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:29:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:29:56.400 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:29:56.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:56.454+0000 7f12f79f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:56.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:29:56.454+0000 7f12f79f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:29:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:56.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:29:56.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:29:56.480 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:29:56Z" 2024-06-15T08:29:56.481 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:29:56.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:01.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:01.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:01.483 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:01.525 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:01.525 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:01.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:01.574+0000 7f7243495a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:01.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:01.574+0000 7f7243495a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:01.598 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:01Z" 2024-06-15T08:30:01.603 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:01.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:06.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:06.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:06.605 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:06.648 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:06.648 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:06.690 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:06.686+0000 7fa5c31f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:06.691 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:06.690+0000 7fa5c31f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.713 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:06.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:06.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:06.717 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:06Z" 2024-06-15T08:30:06.718 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:06.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:30:11.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:11.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:11.720 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:11.763 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:11.763 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:11.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:11.814+0000 7f7ab3a72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:11.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:11.814+0000 7f7ab3a72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:11.841 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:11.841 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:11.841 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:11.841 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:11.841 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:11.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:11.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:11Z" 2024-06-15T08:30:11.846 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:11.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.742181_561.1 2024-06-15T08:30:16.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:16.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:16.848 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:16.889 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:16.889 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:16.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:16.942+0000 7f38c32bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:16.941 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:16.942+0000 7f38c32bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:16.961 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:16.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:16.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:16.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.965 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:16.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:16.966 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:16.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:16.966 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:16Z" 2024-06-15T08:30:16.966 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:16.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:30:21.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:21.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:21.969 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:22.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:22.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:22.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:22.061+0000 7faf11d45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:22.063 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:22.065+0000 7faf11d45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:22.082 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:22.082 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:22.082 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:22.082 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:22.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:22.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:22.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:22.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:22Z" 2024-06-15T08:30:22.087 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:22.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:27.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:27.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:27.089 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:27.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:27.133 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:27.177 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:27.177+0000 7f902a375a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:27.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:27.177+0000 7f902a375a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:27.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:27.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:27.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:27.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:27.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:27Z" 2024-06-15T08:30:27.202 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:27.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:27.609 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:30:27.594+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:30:32.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:32.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:32.204 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:32.250 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:32.250 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:32.292 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:32.293+0000 7fe1ce00aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:32.293 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:32.293+0000 7fe1ce00aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:32.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:32.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:32.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:32.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:32.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:32.316 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:32.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:32.316 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:32Z" 2024-06-15T08:30:32.316 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:32.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:37.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:37.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:37.318 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:37.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:37.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:37.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:37.417+0000 7f1e7bc67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:37.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:37.417+0000 7f1e7bc67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:37.438 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:37.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:37.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:37.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:37.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:37Z" 2024-06-15T08:30:37.443 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:37.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:42.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:42.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:42.445 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:42.499 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:42.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:42.584 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:42.585+0000 7f6d82443a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:42.585 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:42.585+0000 7f6d82443a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:42.605 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:42.605 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:42.605 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:30:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:42Z" 2024-06-15T08:30:42.610 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:42.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:30:47.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:47.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:47.613 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:47.657 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:47.657 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:47.697 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:47.697+0000 7fc04d255a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:47.698 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:47.697+0000 7fc04d255a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:47.716 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:30:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:47.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:47.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:47.722 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:47.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:47.722 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:47Z" 2024-06-15T08:30:47.722 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:47.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:30:52.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:52.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:52.724 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:52.768 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:52.768 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:52.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:52.825+0000 7fc86371aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:52.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:52.825+0000 7fc86371aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:52.843 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:52.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:52.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:52.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:52.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:52Z" 2024-06-15T08:30:52.848 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:52.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:30:57.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:57.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:30:57.850 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:30:57.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:30:57.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:30:57.949 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:57.949+0000 7f1de8f84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:57.950 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:30:57.949+0000 7f1de8f84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:30:57.969 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:30:57.969 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:30:57.969 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:57.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:30:57.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:30:57.972 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:30:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:30:57Z" 2024-06-15T08:30:57.974 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:30:57.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:02.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:02.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:02.976 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:03.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:03.021 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:03.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:03.101+0000 7f14b60aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:03.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:03.101+0000 7f14b60aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:03.122 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:03.123 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:03.124 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:03.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:03Z" 2024-06-15T08:31:03.127 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:03.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:31:08.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:08.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:08.130 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:08.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:08.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:08.216 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:08.216+0000 7efe6978fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:08.217 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:08.216+0000 7efe6978fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:08.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:08.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:08.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:08.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:08Z" 2024-06-15T08:31:08.240 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:08.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:13.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:13.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:13.242 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:13.285 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:13.286 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:13.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:13.328+0000 7f20bb694a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:13.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:13.328+0000 7f20bb694a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:13.347 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:13.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:13.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:13.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:13Z" 2024-06-15T08:31:13.351 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:13.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:18.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:18.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:18.354 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:18.396 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:18.396 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:18.450 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:18.448+0000 7fb9a8e2da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:18.451 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:18.452+0000 7fb9a8e2da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:18.473 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:18.473 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:18.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:18.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:18Z" 2024-06-15T08:31:18.478 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:18.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.615034_551.1 2024-06-15T08:31:23.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:23.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:23.481 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:23.525 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:23.525 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:23.569 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:23.568+0000 7f9e8d5bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:23.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:23.568+0000 7f9e8d5bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:23.591 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:23.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:23.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:23.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:23.595 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:23Z" 2024-06-15T08:31:23.595 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:23.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:28.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:28.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:28.596 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:28.639 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:28.640 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:28.691 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:28.692+0000 7f5055cf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:28.692 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:28.692+0000 7f5055cf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.717 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:28.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:28.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:28Z" 2024-06-15T08:31:28.721 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:28.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:31:33.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:33.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:33.724 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:33.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:33.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:33.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:33.816+0000 7f0230d72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:33.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:33.816+0000 7f0230d72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:33.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:33.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:33Z" 2024-06-15T08:31:33.848 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:33.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:31:38.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:38.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:38.850 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:38.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:38.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:38.939 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:38.940+0000 7fc6fd3d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:38.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:38.940+0000 7fc6fd3d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:38.959 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:38Z" 2024-06-15T08:31:38.964 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:38.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:31:43.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:43.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:43.967 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:44.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:44.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:44.058 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:44.056+0000 7fb0f7a6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:44.059 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:44.060+0000 7fb0f7a6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:44.079 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:44.080 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:44.081 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:44.082 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:44.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:44.084 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:44Z" 2024-06-15T08:31:44.084 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:44.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:49.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:49.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:49.086 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:49.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:49.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:49.215 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:49.216+0000 7f5526434a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:49.216 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:49.216+0000 7f5526434a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:49.241 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:49.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:49.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:49.245 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:49Z" 2024-06-15T08:31:49.246 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:49.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:31:54.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:54.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:54.248 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:54.292 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:54.292 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:54.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:54.336+0000 7f869f292a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:54.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:54.340+0000 7f869f292a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:54.358 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:54.359 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:54.360 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:54.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:54.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:54.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:54.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:54.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.363 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:54.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:54.364 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:54.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:54.364 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:54Z" 2024-06-15T08:31:54.364 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:54.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:31:59.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:59.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:31:59.366 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:31:59.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:31:59.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:31:59.473 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:59.471+0000 7fc589732a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:59.474 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:31:59.471+0000 7fc589732a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.499 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.500 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:31:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:31:59.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:31:59.503 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:31:59.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:31:59.504 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:31:59Z" 2024-06-15T08:31:59.504 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:31:59.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:04.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:04.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:04.507 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:04.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:04.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:04.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:04.603+0000 7f27f2863a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:04.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:04.603+0000 7f27f2863a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:04.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:04.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:04.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:04.629 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:04.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:04.630 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:04Z" 2024-06-15T08:32:04.630 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:04.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:09.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:09.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:09.632 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:09.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:09.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:09.753 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:09.751+0000 7fb23eccaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:09.754 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:09.751+0000 7fb23eccaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:09.802 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:09Z" 2024-06-15T08:32:09.807 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:09.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:14.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:14.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:14.810 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:14.855 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:14.855 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:14.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:14.903+0000 7f467202da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:14.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:14.903+0000 7f467202da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:14.928 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:14.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:14.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:14.932 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:14.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:14.933 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:14Z" 2024-06-15T08:32:14.933 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:14.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:32:19.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:19.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:19.935 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:19.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:19.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:20.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:20.019+0000 7feacee0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:20.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:20.019+0000 7feacee0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:20.038 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:20Z" 2024-06-15T08:32:20.044 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:20.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:32:25.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:25.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:25.046 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:25.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:25.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:25.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:25.143+0000 7ff7eeb7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:25.144 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:25.143+0000 7ff7eeb7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:25.169 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:25.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:25.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:25.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:25.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:25Z" 2024-06-15T08:32:25.174 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:25.175 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:32:30.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:30.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:30.176 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:30.220 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:30.220 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:30.265 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:30.263+0000 7f8e62e96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:30.266 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:30.263+0000 7f8e62e96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:30.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:30.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:30.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:30.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:30Z" 2024-06-15T08:32:30.294 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:30.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:35.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:35.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:35.297 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:35.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:35.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:35.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:35.403+0000 7f7c91923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:35.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:35.403+0000 7f7c91923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:35.430 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:35.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:35.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:35.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:35.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:35.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:35.437 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:35.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:35.438 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:35Z" 2024-06-15T08:32:35.438 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:35.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:40.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:40.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:40.441 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:40.485 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:40.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:40.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:40.543+0000 7f7f21952a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:40.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:40.547+0000 7f7f21952a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:40Z" 2024-06-15T08:32:40.574 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:40.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:45.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:45.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:45.576 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:45.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:45.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:45.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:45.662+0000 7f337ed0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:45.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:45.662+0000 7f337ed0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:45.683 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:45.683 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:45.683 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:45.683 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:45.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:45.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:45Z" 2024-06-15T08:32:45.689 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:45.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:32:50.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:50.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:50.691 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:50.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:50.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:50.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:50.778+0000 7f14ea4eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:50.782 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:50.778+0000 7f14ea4eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:50.803 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:50Z" 2024-06-15T08:32:50.808 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:50.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:32:55.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:55.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:32:55.811 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:32:55.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:32:55.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:32:55.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:55.902+0000 7fc170707a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:55.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:32:55.902+0000 7fc170707a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:32:55.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:32:55Z" 2024-06-15T08:32:55.930 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:32:55.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:00.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:00.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:00.932 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:00.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:00.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:01.035 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:01.030+0000 7f3d1a714a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:01.036 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:01.034+0000 7f3d1a714a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:01.059 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:01Z" 2024-06-15T08:33:01.065 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:01.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:33:06.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:06.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:06.068 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:06.112 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:06.112 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:06.165 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:06.162+0000 7ff0dfb13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:06.166 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:06.162+0000 7ff0dfb13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:06.185 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:06Z" 2024-06-15T08:33:06.190 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:06.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:11.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:11.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:11.193 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:11.236 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:11.236 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:11.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:11.278+0000 7f136e58ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:11.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:11.278+0000 7f136e58ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:11.299 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:11Z" 2024-06-15T08:33:11.304 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:11.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438845.799732_672.1 2024-06-15T08:33:16.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:16.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:16.307 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:16.351 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:16.351 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:16.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:16.402+0000 7eff50d37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:16.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:16.402+0000 7eff50d37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:16.423 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T08:33:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:16Z" 2024-06-15T08:33:16.428 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:16.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438845.799732_672.1 2024-06-15T08:33:21.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:21.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:21.430 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:21.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:21.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:21.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:21.518+0000 7f116b024a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:21.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:21.518+0000 7f116b024a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:21Z" 2024-06-15T08:33:21.546 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:21.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438845.799732_672.1 2024-06-15T08:33:26.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:26.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:26.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:26.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:26.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:26.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:26.642+0000 7f55e2d52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:26.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:26.646+0000 7f55e2d52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:26.670 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:26Z" 2024-06-15T08:33:26.675 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:26.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:33:31.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:31.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:31.678 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:31.776 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:31.778+0000 7f84d77fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:31.777 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:31.778+0000 7f84d77fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:31.797 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:31.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:31.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:31Z" 2024-06-15T08:33:31.802 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:31.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.707204_558.1 2024-06-15T08:33:36.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:36.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:36.804 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:36.848 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:36.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:36.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:36.893+0000 7fbcf6bdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:36.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:36.893+0000 7fbcf6bdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:36.915 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:36.915 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:36.915 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:36.915 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:36.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:36.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:36.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:36.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:36Z" 2024-06-15T08:33:36.920 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:36.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:41.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:41.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:41.923 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:41.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:41.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:42.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:42.017+0000 7f02b6989a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:42.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:42.021+0000 7f02b6989a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:42.038 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:42.038 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:42.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:42.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:42.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:42.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:42Z" 2024-06-15T08:33:42.043 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:42.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:47.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:47.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:47.045 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:47.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:47.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:47.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:47.141+0000 7f73af206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:47.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:47.141+0000 7f73af206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:47.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:47.167 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:47Z" 2024-06-15T08:33:47.168 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:47.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:52.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:52.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:52.171 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:52.220 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:52.220 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:52.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:52.281+0000 7f1d274aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:52.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:52.281+0000 7f1d274aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:52.301 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:52.301 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:52.301 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:52.301 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:52.301 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:52.302 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:52.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:52.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:52.305 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:52Z" 2024-06-15T08:33:52.306 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:52.307 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:33:57.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:57.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:33:57.308 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:33:57.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:33:57.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:33:57.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:57.401+0000 7f555ca27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:57.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:33:57.405+0000 7f555ca27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:33:57.426 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:33:57.426 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:57.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:33:57.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:57.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:57.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:33:57Z" 2024-06-15T08:33:57.431 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:33:57.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.690655_555.1 2024-06-15T08:34:02.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:02.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:02.433 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:02.485 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:02.485 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:02.528 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:02.529+0000 7f65ccc78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:02.529 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:02.529+0000 7f65ccc78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:02.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:02.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:02.552 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:02.553 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:02.553 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:02Z" 2024-06-15T08:34:02.553 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:02.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440139.598802_945.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:07.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-06-15T08:34:07.556 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-58 2024-06-15T08:34:07.607 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-59 2024-06-15T08:34:07.686 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-60 2024-06-15T08:34:08.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:08.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:08.267 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:08.476 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:08.476 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:08.626 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:34:08.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718440447.840466_970.1", 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:34:07.840466Z" 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:34:08.711 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:34:08.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:34:08.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:08.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:08.878 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:08.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:08.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:08.961 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:08.961+0000 7f0281948a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:08.962 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:08.961+0000 7f0281948a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:08.988 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:08.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:08.990 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:08.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:08.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:08Z" 2024-06-15T08:34:08.993 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:08.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:34:13.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:13.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:13.996 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:14.039 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:14.039 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:14.087 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:14.089+0000 7fe2a45a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:14.088 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:14.089+0000 7fe2a45a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:14.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:14.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:14.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:14.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:14Z" 2024-06-15T08:34:14.115 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:14.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:17.644 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:34:17.645+0000 7fcb721b1640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:34:17.645 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:34:17.645+0000 7fcb801cd640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:34:17.766 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:34:17.765+0000 7fcb7e1c9640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:34:17.767 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:34:17.765+0000 7fcb7c1c5640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:34:19.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:19.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:19.117 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:19.162 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:19.166 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:19.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:19.209+0000 7f88d3274a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:19.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:19.209+0000 7f88d3274a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:19.244 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:19.244 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:19.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:19.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:19.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:19.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:19Z" 2024-06-15T08:34:19.249 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:19.250 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:24.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:24.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:24.251 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:24.293 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:24.294 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:24.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:24.357+0000 7fdb9dbaea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:24.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:24.361+0000 7fdb9dbaea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:24.379 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:24.379 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:24.379 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:24.379 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:24.379 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:24.380 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:24.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:34:24.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:24.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:24Z" 2024-06-15T08:34:24.384 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:24.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:34:29.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:29.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:29.386 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:29.429 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:29.430 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:29.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:29.480+0000 7f29c7ca6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:29.482 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:29.480+0000 7f29c7ca6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:29.505 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:29.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:29.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:29.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:29.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:29Z" 2024-06-15T08:34:29.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:29.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:34.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:34.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:34.513 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:34.555 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:34.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:34.599 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:34.600+0000 7f6e34a97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:34.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:34.600+0000 7f6e34a97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:34.618 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:34.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:34.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:34.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:34.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:34.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:34.624 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:34.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:34.624 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:34Z" 2024-06-15T08:34:34.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:34.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.759965_563.1 2024-06-15T08:34:39.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:39.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:39.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:39.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:39.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:39.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:39.720+0000 7f848af33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:39.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:39.720+0000 7f848af33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:39.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:39.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:39.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:39.743 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:39.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:39.745 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:39.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:39.745 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:39Z" 2024-06-15T08:34:39.745 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:39.746 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:44.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:44.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:44.748 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:44.792 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:44.792 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:44.843 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:44.844+0000 7f343bd48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:44.844 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:44.844+0000 7f343bd48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.867 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.868 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:44.870 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:44.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:44.872 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:44Z" 2024-06-15T08:34:44.872 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:44.872 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:34:49.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:49.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:49.874 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:49.916 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:49.916 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:49.966 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:49.964+0000 7fe7bb088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:49.967 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:49.968+0000 7fe7bb088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:49.987 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:49.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:49.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:49.990 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:49.991 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:49.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:49.992 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:49Z" 2024-06-15T08:34:49.992 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:49.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.979596_577.1 2024-06-15T08:34:54.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:54.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:34:54.995 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:34:55.038 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:34:55.038 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:34:55.085 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:55.084+0000 7f1a457c7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:55.086 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:34:55.084+0000 7f1a457c7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:34:55.105 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:55.106 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:34:55.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:55.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:34:55.109 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:34:55Z" 2024-06-15T08:34:55.110 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:34:55.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:35:00.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:00.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:00.112 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:00.152 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:00.153 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:00.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:00.204+0000 7f85d2000a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:00.207 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:00.208+0000 7f85d2000a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:00.230 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:00.231 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:00.232 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:00.233 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:00.234 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:00Z" 2024-06-15T08:35:00.235 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:00.236 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.699165_557.1 2024-06-15T08:35:05.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:05.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:05.237 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:05.280 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:05.280 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:05.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:05.320+0000 7f38f8fbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:05.321 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:05.320+0000 7f38f8fbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.339 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:05.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:05.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:05.342 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:05.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:05Z" 2024-06-15T08:35:05.344 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:05.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:35:10.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:10.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:10.347 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:10.391 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:10.392 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:10.463 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:10.464+0000 7fa197bc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:10.465 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:10.464+0000 7fa197bc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:10.495 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:10.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:10.497 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:10.498 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:10Z" 2024-06-15T08:35:10.499 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:10.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.699165_557.1 2024-06-15T08:35:15.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:15.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:15.501 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:15.544 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:15.544 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:15.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:15.592+0000 7fa6931c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:15.594 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:15.592+0000 7fa6931c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:15.615 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:15.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:15.617 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:35:15.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:15.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:15Z" 2024-06-15T08:35:15.620 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:15.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.699165_557.1 2024-06-15T08:35:20.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:20.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:20.623 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:20.668 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:20.668 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:20.709 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:20.708+0000 7fef5ab73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:20.710 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:20.708+0000 7fef5ab73a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:20.729 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:20.730 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:20.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:20.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:20.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:20Z" 2024-06-15T08:35:20.734 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:20.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:35:25.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:25.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:25.736 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:25.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:25.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:25.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:25.819+0000 7f3662c36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:25.821 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:25.819+0000 7f3662c36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:25.844 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:25.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:25.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:25.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:25.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:25.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:25.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:25.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:25Z" 2024-06-15T08:35:25.849 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:25.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:35:28.852 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:35:28.848+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:35:30.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:30.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:30.851 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:30.895 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:30.895 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:30.945 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:30.943+0000 7f78fc2efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:30.946 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:30.943+0000 7f78fc2efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:30.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:30.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:30.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:30.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:30.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:30.972 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:30Z" 2024-06-15T08:35:30.972 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:30.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:35:35.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:35.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:35.974 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:36.017 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:36.017 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:36.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:36.067+0000 7f2791842a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:36.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:36.067+0000 7f2791842a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:36.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:36.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:36.094 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:36.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:36.095 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:36Z" 2024-06-15T08:35:36.095 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:36.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:35:41.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:41.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:41.096 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:41.140 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:41.140 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:41.188 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:41.187+0000 7f3e165c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:41.189 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:41.187+0000 7f3e165c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:41.210 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:41.210 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:41.210 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:41.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:41.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:41.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:41.214 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:41Z" 2024-06-15T08:35:41.215 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:41.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:35:46.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:46.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:46.217 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:46.260 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:46.260 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:46.307 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:46.303+0000 7fbfeb34ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:46.308 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:46.307+0000 7fbfeb34ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:46.326 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:46.326 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:46.326 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:46.326 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:46.326 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:46.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:46.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:46.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:46.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:46.331 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:46.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:46.331 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:46Z" 2024-06-15T08:35:46.331 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:46.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:35:51.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:51.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:51.333 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:51.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:51.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:51.430 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:51.427+0000 7f97516f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:51.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:51.431+0000 7f97516f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:51.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:51.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:51.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.457 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:51.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:51.458 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:51.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:51.458 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:51Z" 2024-06-15T08:35:51.458 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:51.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:35:56.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:56.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:35:56.461 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:35:56.506 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:35:56.507 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:35:56.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:56.543+0000 7f3c00c00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:56.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:35:56.547+0000 7f3c00c00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:35:56.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:56.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:56.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:35:56.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:35:56.573 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:35:56Z" 2024-06-15T08:35:56.574 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:35:56.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:01.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:01.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:01.576 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:01.633 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:01.633 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:01.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:01.719+0000 7fb24a3fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:01.722 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:01.719+0000 7fb24a3fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:01.746 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:01.746 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:01.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:01.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:01.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:01.750 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:01.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:01.752 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:01.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:01.752 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:01Z" 2024-06-15T08:36:01.752 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:01.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:06.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:06.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:06.754 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:06.799 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:06.799 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:06.846 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:06.843+0000 7fbc73ce5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:06.847 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:06.847+0000 7fbc73ce5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:06.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:06.869 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:06.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:06.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:06.873 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:06.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:06.873 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:06Z" 2024-06-15T08:36:06.873 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:06.874 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:11.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:11.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:11.875 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:11.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:11.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:11.967 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:11.967+0000 7ff1aebdfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:11.968 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:11.967+0000 7ff1aebdfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:11.995 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:11.995 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:11.995 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:11.995 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:11.995 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:11.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:11.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:11.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:11.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:11Z" 2024-06-15T08:36:12.000 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:12.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:17.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:17.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:17.002 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:17.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:17.047 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:17.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:17.098+0000 7ff571a17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:17.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:17.098+0000 7ff571a17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:17.123 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:17.124 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:17.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:17.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:17.127 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:17Z" 2024-06-15T08:36:17.128 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:17.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:22.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:22.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:22.129 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:22.172 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:22.173 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:22.236 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:22.234+0000 7f0413b74a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:22.237 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:22.234+0000 7f0413b74a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.262 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:22.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:22.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:22Z" 2024-06-15T08:36:22.266 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:22.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:27.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:27.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:27.269 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:27.311 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:27.311 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:27.363 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:27.362+0000 7f837a770a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:27.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:27.362+0000 7f837a770a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:27.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:27.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:27.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:27Z" 2024-06-15T08:36:27.387 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:27.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:32.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:32.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:32.390 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:32.434 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:32.434 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:32.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:32.498+0000 7f9e4dcb1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:32.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:32.498+0000 7f9e4dcb1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:32.527 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:32.527 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:32.527 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:32.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:32.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:36:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:32.531 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:32Z" 2024-06-15T08:36:32.532 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:32.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.742181_561.1 2024-06-15T08:36:37.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:37.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:37.534 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:37.578 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:37.578 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:37.632 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:37.630+0000 7f05b63faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:37.633 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:37.630+0000 7f05b63faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:37.653 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:37.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:37.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:37.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:37.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:37Z" 2024-06-15T08:36:37.658 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:37.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:42.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:42.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:42.661 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:42.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:42.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:42.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:42.758+0000 7f149c464a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:42.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:42.758+0000 7f149c464a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.784 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:42.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:42.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:42.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.789 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:42.790 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:42Z" 2024-06-15T08:36:42.790 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:42.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:36:47.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:47.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:47.792 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:47.844 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:47.844 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:47.922 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:47.918+0000 7f7c73393a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:47.923 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:47.918+0000 7f7c73393a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:47.941 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:47.941 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:47.941 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:47.942 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:47.943 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:36:47.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:47.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:47.946 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:47.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:47.947 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:47Z" 2024-06-15T08:36:47.947 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:47.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.699165_557.1 2024-06-15T08:36:52.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:52.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:52.949 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:52.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:52.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:53.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:53.042+0000 7f87f3202a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:53.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:53.042+0000 7f87f3202a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:53.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:53.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:53.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:53.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:53.071 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:53.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:53.072 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:53Z" 2024-06-15T08:36:53.072 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:53.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:36:58.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:58.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:36:58.075 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:36:58.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:36:58.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:36:58.171 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:58.170+0000 7fc940ec8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:58.173 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:36:58.170+0000 7fc940ec8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:58.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:58.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:36:58.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:36:58.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:36:58.196 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:36:58.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:36:58.198 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:36:58Z" 2024-06-15T08:36:58.198 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:36:58.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:37:03.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:03.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:03.200 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:03.243 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:03.243 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:03.284 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:03.286+0000 7f5119bb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:03.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:03.286+0000 7f5119bb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:03.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:03.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:03.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:03.308 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:03.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:03.308 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:03Z" 2024-06-15T08:37:03.308 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:03.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:08.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:08.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:08.310 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:08.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:08.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:08.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:08.406+0000 7fc839d18a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:08.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:08.406+0000 7fc839d18a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:08.429 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:08.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:08.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:08.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:08.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:08Z" 2024-06-15T08:37:08.434 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:08.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:13.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:13.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:13.436 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:13.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:13.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:13.526 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:13.525+0000 7faed19e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:13.527 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:13.529+0000 7faed19e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.545 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:13.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:13.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.549 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:13.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:13.550 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:13.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:13.550 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:13Z" 2024-06-15T08:37:13.550 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:13.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.707204_558.1 2024-06-15T08:37:18.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:18.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:18.552 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:18.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:18.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:18.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:18.649+0000 7f0f55ab8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:18.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:18.649+0000 7f0f55ab8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:18.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:18.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:18.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:18.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:18.673 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:18.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:18.673 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:18Z" 2024-06-15T08:37:18.673 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:18.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:23.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:23.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:23.675 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:23.719 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:23.719 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:23.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:23.785+0000 7f5db4f32a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:23.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:23.785+0000 7f5db4f32a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:23.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:23.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:23.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:23.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:23Z" 2024-06-15T08:37:23.834 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:23.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.759965_563.1 2024-06-15T08:37:28.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:28.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:28.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:28.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:28.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:28.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:28.925+0000 7f731220ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:28.926 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:28.925+0000 7f731220ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:28.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:28.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:28.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:28Z" 2024-06-15T08:37:28.949 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:28.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:33.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:33.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:33.952 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:33.996 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:33.996 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:34.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:34.045+0000 7fc3c6432a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:34.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:34.045+0000 7fc3c6432a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:34.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:34.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:34.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.076 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:34.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:34.077 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:34.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:34.077 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:34Z" 2024-06-15T08:37:34.077 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:34.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:39.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:39.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:39.079 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:39.123 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:39.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:39.188 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:39.189+0000 7f7648f1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:39.189 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:39.189+0000 7f7648f1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:39.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:39.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:39.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:39.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:39Z" 2024-06-15T08:37:39.216 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:39.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:44.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:44.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:44.218 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:44.263 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:44.263 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:44.305 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:44.305+0000 7f86250f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:44.306 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:44.305+0000 7f86250f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:44.324 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:44.324 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:44.324 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:44.324 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:44.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:44.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:44.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:44.328 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:44Z" 2024-06-15T08:37:44.329 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:44.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.707204_558.1 2024-06-15T08:37:49.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:49.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:49.331 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:49.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:49.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:49.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:49.437+0000 7f88ae47aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:49.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:49.441+0000 7f88ae47aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:49.459 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:49.460 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:49.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:49.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:49.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:49Z" 2024-06-15T08:37:49.464 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:49.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.707204_558.1 2024-06-15T08:37:54.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:54.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:54.467 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:54.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:54.510 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:54.565 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:54.565+0000 7fa462e8aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:54.566 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:54.565+0000 7fa462e8aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:54.587 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:54.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:54.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:54.590 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:54.591 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:54Z" 2024-06-15T08:37:54.593 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:54.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:37:59.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:59.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:37:59.596 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:37:59.638 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:37:59.639 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:37:59.705 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:59.705+0000 7f5b9406aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:59.706 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:37:59.705+0000 7f5b9406aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:37:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:37:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:37:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:37:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:37:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:37:59Z" 2024-06-15T08:37:59.737 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:37:59.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:38:04.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:04.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:04.739 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:04.782 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:04.782 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:04.834 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:04.833+0000 7fa964025a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:04.834 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:04.833+0000 7fa964025a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:04.854 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:04.855 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:04.856 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:04Z" 2024-06-15T08:38:04.857 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:04.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:38:09.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:09.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:09.862 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:09.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:09.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:09.953 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:09.952+0000 7ff303e54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:09.954 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:09.952+0000 7ff303e54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.083531_582.1", 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.083531Z", 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:09Z" 2024-06-15T08:38:09.979 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:09.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438539.083531_582.1 2024-06-15T08:38:14.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:14.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:14.982 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:15.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:15.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:15.087 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:15.088+0000 7f9f64d1ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:15.088 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:15.088+0000 7f9f64d1ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:15.113 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:15.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:15.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.083531_582.1", 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:15.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.083531Z", 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:15.117 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:15Z" 2024-06-15T08:38:15.118 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:15.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438539.083531_582.1 2024-06-15T08:38:20.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:20.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:20.120 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:20.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:20.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:20.218 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:20.216+0000 7f622fda1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:20.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:20.220+0000 7f622fda1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:20.241 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:20.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:20.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:20Z" 2024-06-15T08:38:20.246 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:20.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:38:25.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:25.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:25.249 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:25.291 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:25.291 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:25.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:25.344+0000 7f4cb09f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:25.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:25.348+0000 7f4cb09f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:25.372 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:25.375 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:25.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.377 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:25.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:25.378 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:25.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:25.378 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:25Z" 2024-06-15T08:38:25.378 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:25.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:38:30.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:30.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:30.380 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:30.425 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:30.425 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:30.478 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:30.476+0000 7fb53c94da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:30.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:30.480+0000 7fb53c94da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:30.505 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:30.505 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:30.505 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:30.505 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:30.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:30.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:30.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:30Z" 2024-06-15T08:38:30.509 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:30.510 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.728887_559.1 2024-06-15T08:38:35.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:35.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:35.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:35.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:35.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:35.599 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:35.600+0000 7f73c7877a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:35.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:35.600+0000 7f73c7877a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:35.618 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:35.618 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:35.618 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:35.618 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:35.618 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:35.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:35.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:35.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:35Z" 2024-06-15T08:38:35.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:35.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.742181_561.1 2024-06-15T08:38:40.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:40.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:40.626 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:40.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:40.677 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:40.727 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:40.728+0000 7ff6e2696a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:40.728 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:40.728+0000 7ff6e2696a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:40.746 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:40.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:40.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:40.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.750 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:40.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:40.751 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:40.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:40.751 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:40Z" 2024-06-15T08:38:40.751 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:40.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:38:45.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:45.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:45.753 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:45.796 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:45.797 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:45.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:45.840+0000 7f1eeac5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:45.840 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:45.840+0000 7f1eeac5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:45.862 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:45.862 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:45.862 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:45.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:45.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:45.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:45.866 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:45Z" 2024-06-15T08:38:45.867 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:45.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:38:50.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:50.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:50.869 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:50.914 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:50.914 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:50.958 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:50.956+0000 7fc2b980da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:50.959 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:50.960+0000 7fc2b980da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:50.979 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:50.979 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:50.980 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:50.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:50.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:50.983 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:50Z" 2024-06-15T08:38:50.984 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:50.985 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.707204_558.1 2024-06-15T08:38:55.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:55.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:38:55.987 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:38:56.038 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:38:56.038 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:38:56.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:56.092+0000 7f7ab258ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:56.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:38:56.092+0000 7f7ab258ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:38:56.114 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:38:56.114 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:38:56.114 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:38:56.114 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:38:56.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:38:56.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:56.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:38:56.118 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:38:56Z" 2024-06-15T08:38:56.119 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:38:56.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.699165_557.1 2024-06-15T08:39:01.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:01.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:01.121 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:01.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:01.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:01.215 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:01.215+0000 7fdc22b67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:01.216 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:01.215+0000 7fdc22b67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:01.236 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:01.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:01.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.747432_562.1", 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.747432Z", 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:01.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:01.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:01.242 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:01Z" 2024-06-15T08:39:01.242 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:01.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.747432_562.1 2024-06-15T08:39:06.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:06.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:06.244 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:06.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:06.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:06.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:06.335+0000 7f46b2e93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:06.336 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:06.335+0000 7f46b2e93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:06.354 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:06.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.747432_562.1", 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.747432Z", 2024-06-15T08:39:06.357 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:06.358 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:06Z" 2024-06-15T08:39:06.359 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:06.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.747432_562.1 2024-06-15T08:39:11.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:11.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:11.362 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:11.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:11.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:11.458 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:11.455+0000 7fd70c5b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:11.459 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:11.459+0000 7fd70c5b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:11.478 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:11.478 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:11.478 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:11.478 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:11.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:11.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:39:11.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:11.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:11Z" 2024-06-15T08:39:11.483 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:11.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440447.840466_970.1 target=1_1718438536.690655_555.1 2024-06-15T08:39:16.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-06-15T08:39:16.486 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-61 2024-06-15T08:39:16.535 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-62 2024-06-15T08:39:16.619 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-63 2024-06-15T08:39:17.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:17.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:17.219 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:17.305 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:17.368 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:39:17.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718440756.664316_1031.1", 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:39:16.664316Z" 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:39:17.404 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:39:17.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:39:17.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:17.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:17.405 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:17.448 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:17.448 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:17.497 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:17.495+0000 7fae49bb3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:17.498 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:17.495+0000 7fae49bb3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:17.517 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:17.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:17.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:17.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:17.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:17Z" 2024-06-15T08:39:17.522 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:17.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:39:17.794 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:39:17.790+0000 7fcb799c0640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:39:17.794 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:39:17.790+0000 7fcb789be640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:39:18.837 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:39:18.834+0000 7fcb749b6640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:39:22.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:22.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:22.525 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:22.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:22.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:22.628 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:22.627+0000 7ff0bcfa4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:22.629 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:22.627+0000 7ff0bcfa4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:22.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:22.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:22.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:22Z" 2024-06-15T08:39:22.653 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:22.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:39:27.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:27.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:27.656 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:27.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:27.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:27.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:27.763+0000 7f28e0459a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:27.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:27.763+0000 7f28e0459a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:27.812 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:27.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:27.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:27Z" 2024-06-15T08:39:27.817 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:27.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:39:32.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:32.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:32.819 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:32.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:32.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:32.908 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:32.907+0000 7f28d56c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:32.909 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:32.907+0000 7f28d56c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:32.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:32.933 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:32.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:32.933 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:32Z" 2024-06-15T08:39:32.933 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:32.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:39:37.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:37.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:37.935 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:37.986 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:37.986 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:38.037 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:38.035+0000 7f0107ceea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:38.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:38.035+0000 7f0107ceea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.058 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:38.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:38.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:38.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.062 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:38.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:38.063 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:38.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:38.063 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:38Z" 2024-06-15T08:39:38.063 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:38.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:39:43.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:43.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:43.066 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:43.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:43.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:43.158 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:43.154+0000 7f16bf68aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:43.159 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:43.158+0000 7f16bf68aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:43.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:43.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:43.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:43.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:43Z" 2024-06-15T08:39:43.184 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:43.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.707204_558.1 2024-06-15T08:39:48.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:48.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:48.186 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:48.241 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:48.241 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:48.292 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:48.290+0000 7fb77d20fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:48.292 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:48.290+0000 7fb77d20fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:48.320 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:48.321 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:48.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:48.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:48.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:48.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:48Z" 2024-06-15T08:39:48.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:48.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.615034_551.1 2024-06-15T08:39:53.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:53.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:53.328 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:53.373 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:53.373 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:53.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:53.422+0000 7f5bb178ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:53.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:53.422+0000 7f5bb178ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:53.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:53.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:53Z" 2024-06-15T08:39:53.447 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:53.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.615034_551.1 2024-06-15T08:39:58.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:58.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:39:58.449 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:39:58.491 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:39:58.491 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:39:58.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:58.538+0000 7f4d0e367a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:58.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:39:58.542+0000 7f4d0e367a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:39:58.563 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:39:58.563 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:39:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:58.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:58.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:39:58.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:39:58Z" 2024-06-15T08:39:58.568 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:39:58.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:40:03.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:03.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:03.570 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:03.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:03.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:03.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:03.662+0000 7f0a456d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:03.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:03.666+0000 7f0a456d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:03.685 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:03.685 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:03.685 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:03.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:03.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:03.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:03.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:03Z" 2024-06-15T08:40:03.690 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:03.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:40:08.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:08.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:08.693 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:08.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:08.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:08.778 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:08.774+0000 7ff96cb4ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:08.779 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:08.778+0000 7ff96cb4ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:08.802 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:08.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:08.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:08.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:08.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:08Z" 2024-06-15T08:40:08.807 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:08.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:13.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:13.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:13.810 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:13.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:13.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:13.945 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:13.942+0000 7fa6a1475a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:13.946 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:13.942+0000 7fa6a1475a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:13.964 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:13.964 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:13Z" 2024-06-15T08:40:13.970 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:13.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:18.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:18.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:18.973 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:19.016 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:19.016 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:19.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:19.066+0000 7f4c9f681a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:19.065 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:19.066+0000 7f4c9f681a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:19.083 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:19.083 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:19.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:19.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:19.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:19.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:19Z" 2024-06-15T08:40:19.088 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:19.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:24.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:24.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:24.091 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:24.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:24.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:24.184 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:24.186+0000 7f42fb058a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:24.185 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:24.186+0000 7f42fb058a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:24.204 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:24.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:24.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.208 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:24.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:24.209 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:24.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:24.209 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:24Z" 2024-06-15T08:40:24.209 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:24.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:29.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:29.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:29.211 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:29.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:29.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:29.305 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:29.305+0000 7f94845d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:29.306 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:29.305+0000 7f94845d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:29.324 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:29.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:29.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:29.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:29.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:29.329 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:29Z" 2024-06-15T08:40:29.329 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:29.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:34.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:34.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:34.331 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:34.374 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:34.374 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:34.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:34.417+0000 7fc9595cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:34.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:34.417+0000 7fc9595cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:34.436 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:34.436 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:34.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:34.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:34.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:34.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:34Z" 2024-06-15T08:40:34.441 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:34.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:39.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:39.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:39.444 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:39.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:39.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:39.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:39.541+0000 7ff12ff27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:39.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:39.545+0000 7ff12ff27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:39.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:39.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:39.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:39Z" 2024-06-15T08:40:39.571 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:39.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:40:40.089 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:40:40.086+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:40:44.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:44.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:44.573 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:44.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:44.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:44.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:44.661+0000 7f61daa09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:44.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:44.665+0000 7f61daa09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:44.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:44.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:44.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:44.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:44.686 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:44.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:44.687 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:44Z" 2024-06-15T08:40:44.687 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:44.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:49.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:49.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:49.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:49.732 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:49.732 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:49.796 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:49.797+0000 7fbbbeaf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:49.796 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:49.797+0000 7fbbbeaf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:49.817 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:49.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:49.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:49.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:49.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:49Z" 2024-06-15T08:40:49.822 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:49.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:54.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:54.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:54.825 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:54.868 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:54.869 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:40:54.915 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:54.913+0000 7f3d6fc5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:54.916 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:40:54.917+0000 7f3d6fc5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:54.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:54.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:54.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:40:54.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:40:54Z" 2024-06-15T08:40:54.940 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:40:54.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:40:59.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:59.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:40:59.942 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:40:59.986 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:40:59.986 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:00.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:00.033+0000 7fbe7548aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:00.032 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:00.033+0000 7fbe7548aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:00.058 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:00.058 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:00.058 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:00.059 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:00.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:00.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:00.062 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:00Z" 2024-06-15T08:41:00.063 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:00.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.707204_558.1 2024-06-15T08:41:05.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:05.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:05.065 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:05.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:05.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:05.157 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:05.157+0000 7fa57f3f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:05.158 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:05.157+0000 7fa57f3f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.184 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:05.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:05.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:05.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:05.188 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:05.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:05.189 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:05Z" 2024-06-15T08:41:05.189 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:05.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:41:10.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:10.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:10.191 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:10.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:10.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:10.296 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:10.297+0000 7efe64a91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:10.297 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:10.297+0000 7efe64a91a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:10.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:10.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:10.325 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:10.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:10Z" 2024-06-15T08:41:10.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:10.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438537.989943_579.1 2024-06-15T08:41:15.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:15.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:15.328 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:15.369 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:15.369 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:15.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:15.420+0000 7f08e3d96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:15.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:15.420+0000 7f08e3d96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:15.442 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:15.442 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:15.442 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:15.442 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:15.442 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:15.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:15.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:15.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:15Z" 2024-06-15T08:41:15.447 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:15.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.699165_557.1 2024-06-15T08:41:20.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:20.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:20.450 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:20.493 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:20.493 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:20.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:20.540+0000 7fc9724e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:20.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:20.540+0000 7fc9724e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:20.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:20.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:20.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:20.567 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:20Z" 2024-06-15T08:41:20.567 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:20.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:41:25.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:25.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:25.569 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:25.612 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:25.612 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:25.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:25.664+0000 7f7d933aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:25.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:25.664+0000 7f7d933aaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:25.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:25.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:25.691 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:25.692 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:25Z" 2024-06-15T08:41:25.692 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:25.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:41:30.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:30.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:30.694 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:30.737 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:30.737 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:30.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:30.788+0000 7ff5d61bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:30.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:30.788+0000 7ff5d61bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:30.812 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:30.812 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:30.812 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:30.812 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:30.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:30.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:30.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:30.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:30Z" 2024-06-15T08:41:30.817 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:30.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:41:35.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:35.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:35.822 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:35.864 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:35.864 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:35.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:35.936+0000 7fe05e3aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:35.939 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:35.940+0000 7fe05e3aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:35.961 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:35.961 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:35.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:35.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:35.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:35.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:35.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:35Z" 2024-06-15T08:41:35.967 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:35.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:41:40.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:40.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:40.969 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:41.012 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:41.012 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:41.054 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:41.052+0000 7f56ca15da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:41.055 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:41.052+0000 7f56ca15da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:41.133 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:41.133 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:41.133 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:41.133 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:41.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:41.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:41Z" 2024-06-15T08:41:41.138 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:41.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:41:46.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:46.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:46.141 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:46.185 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:46.185 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:46.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:46.228+0000 7fc4dc5dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:46.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:46.228+0000 7fc4dc5dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:46.247 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:46.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:46.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:46.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:46.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:46Z" 2024-06-15T08:41:46.252 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:46.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:41:51.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:51.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:51.254 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:51.297 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:51.297 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:51.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:51.340+0000 7f8da1df6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:51.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:51.340+0000 7f8da1df6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:51.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:51.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:51.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:51.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:51Z" 2024-06-15T08:41:51.365 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:51.366 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.699165_557.1 2024-06-15T08:41:56.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:56.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:41:56.367 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:41:56.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:41:56.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:41:56.449 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:56.448+0000 7f9205756a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:56.450 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:41:56.448+0000 7f9205756a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.471 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:56.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:41:56.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:41:56.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:41:56.476 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:41:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:41:56.477 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:41:56Z" 2024-06-15T08:41:56.477 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:41:56.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:42:01.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:01.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:01.479 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:01.523 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:01.523 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:01.576 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:01.575+0000 7f8870d49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:01.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:01.575+0000 7f8870d49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:01.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.598 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:01.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:01.600 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:01.601 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:01.602 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:01Z" 2024-06-15T08:42:01.602 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:01.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:06.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:06.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:06.604 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:06.647 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:06.647 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:06.701 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:06.699+0000 7fa392684a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:06.702 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:06.699+0000 7fa392684a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:06.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:06.727 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:06.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:06.727 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:06Z" 2024-06-15T08:42:06.727 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:06.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:42:11.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:11.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:11.729 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:11.772 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:11.772 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:11.836 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:11.835+0000 7f2a83a34a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:11.837 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:11.835+0000 7f2a83a34a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:11.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:11.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:11.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:11.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:11.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:11Z" 2024-06-15T08:42:11.865 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:11.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:16.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:16.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:16.866 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:16.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:16.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:16.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:16.967+0000 7f0db8985a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:16.972 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:16.971+0000 7f0db8985a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:16Z" 2024-06-15T08:42:17.002 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:17.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:22.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:22.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:22.004 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:22.047 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:22.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:22.103+0000 7fe7496e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:22.104 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:22.103+0000 7fe7496e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:22.130 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:22.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:22.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:22.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:22.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:22.135 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:22.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:22.136 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:22Z" 2024-06-15T08:42:22.136 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:22.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:27.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:27.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:27.138 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:27.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:27.182 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:27.243 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:27.239+0000 7f8a2fe2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:27.244 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:27.243+0000 7f8a2fe2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:27.270 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:27.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:27.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:27.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:27.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.275 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:27.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:27.276 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:27.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:27.276 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:27Z" 2024-06-15T08:42:27.276 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:27.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438538.004281_580.1 2024-06-15T08:42:32.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:32.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:32.278 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:32.321 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:32.321 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:32.376 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:32.375+0000 7f303395ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:32.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:32.375+0000 7f303395ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.397 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:32.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:32.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:32.400 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:32Z" 2024-06-15T08:42:32.401 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:32.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:37.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:37.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:37.403 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:37.446 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:37.446 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:37.490 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:37.487+0000 7fde0ef65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:37.491 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:37.487+0000 7fde0ef65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:37.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:37.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:37.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:37.515 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:37Z" 2024-06-15T08:42:37.516 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:37.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:42:42.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:42.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:42.517 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:42.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:42.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:42.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:42.627+0000 7f26b4dc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:42.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:42.631+0000 7f26b4dc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.665 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:42Z" 2024-06-15T08:42:42.670 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:42.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:47.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:47.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:47.672 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:47.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:47.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:47.758 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:47.754+0000 7f05af3e3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:47.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:47.754+0000 7f05af3e3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:47.778 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:47.778 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:47.778 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:47.778 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:47.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:47.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:47.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.782 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:47.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:47.783 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:47.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:47.783 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:47Z" 2024-06-15T08:42:47.783 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:47.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:42:52.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:52.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:52.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:52.835 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:52.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:52.877 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:52.874+0000 7f49426c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:52.878 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:52.874+0000 7f49426c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:52.895 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:52.895 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:52.895 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:52.895 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:52.895 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:52.896 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:52.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:52.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:52.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:52.900 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:52Z" 2024-06-15T08:42:52.901 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:52.901 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.707204_558.1 2024-06-15T08:42:57.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:57.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:42:57.903 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:42:57.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:42:57.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:42:58.000 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:57.998+0000 7f7732267a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:58.001 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:42:57.998+0000 7f7732267a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:58.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:42:58.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:58.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:42:58.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:42:58Z" 2024-06-15T08:42:58.025 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:42:58.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:43:03.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:03.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:03.028 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:03.073 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:03.073 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:03.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:03.122+0000 7fef6d0c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:03.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:03.122+0000 7fef6d0c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:03.143 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:03.143 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:03.143 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:03.143 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:03.143 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:03.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:03.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:03.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:03.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:03.148 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:03Z" 2024-06-15T08:43:03.148 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:03.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:43:08.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:08.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:08.150 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:08.192 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:08.193 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:08.234 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:08.230+0000 7fd84e48aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:08.235 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:08.230+0000 7fd84e48aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:08.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:08.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:08.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:08.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:08.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:08.265 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:08Z" 2024-06-15T08:43:08.265 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:08.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438539.163871_589.1 2024-06-15T08:43:13.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:13.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:13.267 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:13.309 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:13.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:13.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:13.366+0000 7f2991923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:13.369 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:13.366+0000 7f2991923a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:13.394 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:13.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:13.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:13.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:13.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:13.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:13.399 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:13Z" 2024-06-15T08:43:13.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:13.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:43:18.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:18.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:18.402 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:18.447 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:18.447 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:18.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:18.506+0000 7f9b01390a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:18.510 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:18.506+0000 7f9b01390a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:18.531 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:18.531 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:18.531 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:18.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:18.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:18.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:18Z" 2024-06-15T08:43:18.536 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:18.537 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:43:23.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:23.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:23.538 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:23.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:23.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:23.637 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:23.638+0000 7f6eeb26ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:23.638 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:23.638+0000 7f6eeb26ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.664 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:23.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:23.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:23.668 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:23.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:23.669 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:23Z" 2024-06-15T08:43:23.669 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:23.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.742181_561.1 2024-06-15T08:43:28.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:28.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:28.670 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:28.713 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:28.713 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:28.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:28.774+0000 7f978dc79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:28.773 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:28.774+0000 7f978dc79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:28.799 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:28.799 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:28.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:28Z" 2024-06-15T08:43:28.804 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:28.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:43:33.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:33.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:33.806 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:33.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:33.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:33.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:33.897+0000 7fbfd49c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:33.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:33.901+0000 7fbfd49c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:33.918 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:33.918 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:33.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:33.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:33.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:33Z" 2024-06-15T08:43:33.924 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:33.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.742181_561.1 2024-06-15T08:43:38.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:38.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:38.927 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:38.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:38.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:39.054 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:39.053+0000 7fe51d904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:39.054 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:39.053+0000 7fe51d904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:39.072 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:39.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:39Z" 2024-06-15T08:43:39.078 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:39.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.699165_557.1 2024-06-15T08:43:44.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:44.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:44.080 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:44.122 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:44.122 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:44.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:44.185+0000 7f5fdfea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:44.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:44.185+0000 7f5fdfea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:44.212 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:44.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:44.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:43:44.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:44.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:44Z" 2024-06-15T08:43:44.217 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:44.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:43:49.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:49.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:49.220 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:49.263 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:49.263 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:49.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:49.313+0000 7f14b978ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:49.316 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:49.317+0000 7f14b978ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:49.337 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:49.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:49.337 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:49.337 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:49.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:49.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:43:49.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:49.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:49Z" 2024-06-15T08:43:49.342 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:49.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:43:54.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:54.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:54.344 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:54.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:54.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:54.451 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:54.449+0000 7ff4f7602a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:54.452 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:54.453+0000 7ff4f7602a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:54.477 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:54.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:54.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:54.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:54.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:54Z" 2024-06-15T08:43:54.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:54.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:43:59.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:59.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:43:59.487 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:43:59.540 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:43:59.540 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:43:59.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:59.601+0000 7fd726ed9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:59.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:43:59.601+0000 7fd726ed9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:43:59.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:59.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:43:59.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:43:59.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:59.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:43:59Z" 2024-06-15T08:43:59.626 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:43:59.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:44:04.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:04.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:04.628 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:04.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:04.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:04.727 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:04.725+0000 7f1adfcffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:04.728 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:04.729+0000 7f1adfcffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:04.753 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:04.753 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:04.753 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:04.753 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:04.753 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:04.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:04.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:04.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:04.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:04Z" 2024-06-15T08:44:04.758 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:04.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:44:09.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:09.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:09.760 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:09.804 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:09.804 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:09.856 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:09.857+0000 7fcb26ab7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:09.856 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:09.857+0000 7fcb26ab7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:09.876 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:09.876 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:09.876 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:09.876 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:09.876 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:09.877 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:09.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:09.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:09.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:09.881 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:09Z" 2024-06-15T08:44:09.882 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:09.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.690655_555.1 2024-06-15T08:44:14.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:14.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:14.883 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:14.927 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:14.927 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:14.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:14.969+0000 7fe52ae6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:14.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:14.969+0000 7fe52ae6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:14.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:14Z" 2024-06-15T08:44:14.999 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:15.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.728887_559.1 2024-06-15T08:44:20.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:20.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:20.001 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:20.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:20.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:20.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:20.096+0000 7fe414456a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:20.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:20.096+0000 7fe414456a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.903214_570.1", 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.903214Z", 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:20.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:20.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:20.129 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:20.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:20.129 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:20Z" 2024-06-15T08:44:20.129 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:20.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718440756.664316_1031.1 target=1_1718438536.903214_570.1 2024-06-15T08:44:25.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-06-15T08:44:25.132 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-64 2024-06-15T08:44:25.186 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xdvrxv-65 2024-06-15T08:44:25.271 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xdvrxv-66 2024-06-15T08:44:27.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:27.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:27.069 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:27.122 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:27.123 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:27.164 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:44:27.180 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:44:27.180 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:44:27.180 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:27.180 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718441065.379479_1088.1", 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:44:25.379479Z" 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:27.181 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:44:27.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:44:27.182 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:44:27.183 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:44:27.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:27.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:27.184 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:27.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:27.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:27.290 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:27.288+0000 7f555263ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:27.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:27.288+0000 7f555263ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:27.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.318 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:27.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:27.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:27.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:27.322 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:27Z" 2024-06-15T08:44:27.322 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:27.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:44:32.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:32.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:32.324 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:32.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:32.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:32.414 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:32.412+0000 7f2352f14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:32.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:32.412+0000 7f2352f14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:32.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:32.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:32.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:32.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:32Z" 2024-06-15T08:44:32.440 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:32.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:44:37.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:37.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:37.443 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:37.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:37.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:37.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:37.544+0000 7fd6b5db7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:37.548 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:37.548+0000 7fd6b5db7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:37.570 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:37.570 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:37.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:37.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:44:37.573 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:37.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:37Z" 2024-06-15T08:44:37.575 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:37.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:44:37.987 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:44:37.984+0000 7fcb729b2640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:44:42.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:42.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:42.578 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:42.621 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:42.621 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:42.674 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:42.672+0000 7fefd6f82a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:42.674 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:42.672+0000 7fefd6f82a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:42.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:42.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:42.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:42.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:42Z" 2024-06-15T08:44:42.697 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:42.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:44:47.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:47.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:47.700 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:47.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:47.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:47.787 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:47.784+0000 7f558a4c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:47.787 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:47.788+0000 7f558a4c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.807 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:47.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:47.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:47.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:47.811 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:47Z" 2024-06-15T08:44:47.812 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:47.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438845.752631_668.1 2024-06-15T08:44:52.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:52.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:52.814 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:52.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:52.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:52.910 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:52.908+0000 7f682ff94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:52.911 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:52.908+0000 7f682ff94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:52.936 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:52.936 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:52.936 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:52.936 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:52.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:52.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:44:52.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:52.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:52Z" 2024-06-15T08:44:52.941 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:52.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:44:57.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:57.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:44:57.946 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:44:57.991 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:44:57.991 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:44:58.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:58.044+0000 7f1206c57a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:58.047 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:44:58.044+0000 7f1206c57a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:44:58.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:58.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:58.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:44:58.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:44:58Z" 2024-06-15T08:44:58.076 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:44:58.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:03.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:03.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:03.079 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:03.123 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:03.123 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:03.176 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:03.175+0000 7f12aa020a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:03.177 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:03.175+0000 7f12aa020a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:03.200 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:03.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:03Z" 2024-06-15T08:45:03.205 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:03.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:08.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:08.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:08.208 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:08.251 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:08.251 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:08.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:08.299+0000 7f524ad71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:08.301 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:08.299+0000 7f524ad71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:08.321 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:08.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:08.321 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:08.321 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:08.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:08.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:45:08.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:08.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:08Z" 2024-06-15T08:45:08.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:08.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:45:13.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:13.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:13.328 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:13.372 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:13.372 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:13.414 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:13.411+0000 7fc6754d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:13.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:13.411+0000 7fc6754d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:13.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:13.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.438 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:13.439 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:13Z" 2024-06-15T08:45:13.439 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:13.440 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:18.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:18.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:18.441 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:18.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:18.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:18.552 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:18.551+0000 7ff8c75e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:18.553 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:18.551+0000 7ff8c75e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:18.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:18.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:18.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:18.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:18.583 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:18.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:18.584 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:18Z" 2024-06-15T08:45:18.584 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:18.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:45:23.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:23.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:23.586 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:23.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:23.630 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:23.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:23.667+0000 7f24a0160a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:23.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:23.671+0000 7f24a0160a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:23.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:23.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:23.695 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:23Z" 2024-06-15T08:45:23.696 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:23.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:28.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:28.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:28.698 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:28.741 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:28.741 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:28.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:28.787+0000 7f0119d59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:28.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:28.787+0000 7f0119d59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:28.808 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:28Z" 2024-06-15T08:45:28.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:28.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:33.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:33.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:33.816 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:33.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:33.862 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:33.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:33.903+0000 7f9f9bb2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:33.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:33.903+0000 7f9f9bb2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:33.933 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:33.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:33.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:33.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:33.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:33Z" 2024-06-15T08:45:33.938 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:33.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:45:38.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:38.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:38.941 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:38.985 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:38.985 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:39.026 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:39.023+0000 7f8ae5abba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:39.028 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:39.027+0000 7f8ae5abba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:39.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:39.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:39.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:39.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:39Z" 2024-06-15T08:45:39.073 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:39.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:40.101 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:45:40.099+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:45:44.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:44.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:44.076 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:44.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:44.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:44.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:44.159+0000 7f4cc7912a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:44.163 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:44.159+0000 7f4cc7912a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:44.181 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:44.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:44.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:44.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:44Z" 2024-06-15T08:45:44.186 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:44.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:49.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:49.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:49.188 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:49.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:49.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:49.272 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:49.271+0000 7ff71dfcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:49.273 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:49.271+0000 7ff71dfcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:49.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:49.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:49.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:49Z" 2024-06-15T08:45:49.296 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:49.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:45:54.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:54.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:54.298 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:54.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:54.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:54.387 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:54.382+0000 7f0e2b308a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:54.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:54.386+0000 7f0e2b308a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:54.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:54Z" 2024-06-15T08:45:54.412 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:54.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:45:59.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:59.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:45:59.414 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:45:59.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:45:59.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:45:59.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:59.510+0000 7f2c70b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:59.516 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:45:59.514+0000 7f2c70b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:45:59.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:59.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:45:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:45:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:45:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:45:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:45:59Z" 2024-06-15T08:45:59.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:45:59.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:46:04.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:04.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:04.549 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:04.597 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:04.597 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:04.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:04.638+0000 7f70d3069a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:04.643 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:04.638+0000 7f70d3069a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:04.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:04.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:04.664 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:04.665 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:04Z" 2024-06-15T08:46:04.666 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:04.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:46:09.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:09.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:09.668 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:09.712 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:09.712 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:09.753 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:09.750+0000 7f64f1327a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:09.754 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:09.750+0000 7f64f1327a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:09.771 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:09.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:09.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:09.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:09.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:09Z" 2024-06-15T08:46:09.776 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:09.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:46:14.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:14.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:14.779 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:14.822 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:14.822 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:14.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:14.874+0000 7fa6b7924a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:14.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:14.878+0000 7fa6b7924a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:14.899 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:14.899 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:14.899 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:14.899 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:14.899 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:14.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:14.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:14.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:14.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:14Z" 2024-06-15T08:46:14.904 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:14.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:46:19.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:19.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:19.906 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:19.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:19.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:19.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:19.990+0000 7f46d29f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:19.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:19.990+0000 7f46d29f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:20.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:20.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:20.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:20.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:20.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:19Z" 2024-06-15T08:46:20.017 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:20.018 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.707204_558.1 2024-06-15T08:46:25.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:25.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:25.019 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:25.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:25.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:25.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:25.098+0000 7f27aaca4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:25.104 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:25.102+0000 7f27aaca4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:25.123 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:25.123 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:25.123 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:25.123 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:25Z" 2024-06-15T08:46:25.128 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:25.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:46:30.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:30.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:30.130 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:30.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:30.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:30.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:30.246+0000 7f1d0104fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:30.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:30.246+0000 7f1d0104fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:30.271 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:30.271 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:30.271 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:30.271 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:30.271 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:30.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:30.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:30.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:30.276 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:30.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:30.276 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:30Z" 2024-06-15T08:46:30.276 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:30.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438845.799732_672.1 2024-06-15T08:46:35.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:35.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:35.278 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:35.321 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:35.321 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:35.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:35.370+0000 7fb2c6244a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:35.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:35.374+0000 7fb2c6244a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:35Z" 2024-06-15T08:46:35.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:35.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:46:40.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:40.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:40.402 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:40.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:40.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:40.553 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:40.554+0000 7ff59cd59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:40.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:40.554+0000 7ff59cd59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:40.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:40.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:40.578 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:40Z" 2024-06-15T08:46:40.579 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:40.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:46:45.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:45.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:45.581 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:45.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:45.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:45.678 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:45.677+0000 7fcb4e8e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:45.679 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:45.677+0000 7fcb4e8e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:45.699 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.703 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:45.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:45.704 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:45.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:45.704 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:45Z" 2024-06-15T08:46:45.704 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:45.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.742181_561.1 2024-06-15T08:46:50.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:50.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:50.706 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:50.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:50.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:50.788 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:50.789+0000 7fa961870a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:50.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:50.789+0000 7fa961870a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:50.812 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:46:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:50.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:50Z" 2024-06-15T08:46:50.818 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:50.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:46:55.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:55.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:46:55.822 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:46:55.866 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:46:55.866 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:46:55.926 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:55.925+0000 7f5ba48d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:55.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:46:55.925+0000 7f5ba48d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:46:55.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:46:55.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:46:55.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:55.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:46:55.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:46:55Z" 2024-06-15T08:46:55.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:46:55.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:00.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:00.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:00.959 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:01.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:01.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:01.066 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:01.065+0000 7f04cde61a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:01.067 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:01.065+0000 7f04cde61a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:01.091 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:01.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:01.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:01.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:01Z" 2024-06-15T08:47:01.096 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:01.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:06.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:06.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:06.098 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:06.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:06.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:06.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:06.193+0000 7fb73f301a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:06.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:06.197+0000 7fb73f301a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:06.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:06.222 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:06Z" 2024-06-15T08:47:06.222 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:06.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:11.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:11.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:11.224 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:11.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:11.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:11.309 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:11.309+0000 7f5a1cbf1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:11.310 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:11.309+0000 7f5a1cbf1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:11.333 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:11.334 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:11.335 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:11.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:11.337 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:11Z" 2024-06-15T08:47:11.338 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:11.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:16.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:16.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:16.341 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:16.385 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:16.386 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:16.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:16.441+0000 7fe867754a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:16.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:16.445+0000 7fe867754a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.492 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:16.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:16.495 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:16.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:16.497 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:16Z" 2024-06-15T08:47:16.497 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:16.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:21.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:21.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:21.499 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:21.543 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:21.543 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:21.587 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:21.585+0000 7fe7e239ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:21.588 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:21.589+0000 7fe7e239ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:21.608 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:21.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:21.610 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:21.611 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:21.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:21.613 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:21.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:21.614 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:21Z" 2024-06-15T08:47:21.614 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:21.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:26.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:26.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:26.617 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:26.669 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:26.669 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:26.715 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:26.713+0000 7faa36b52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:26.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:26.717+0000 7faa36b52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:26.740 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:26.740 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:26.740 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:26.740 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:26.740 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:26.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:26.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:26.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.734016_560.1", 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.734016Z", 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:26.744 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:26.745 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.746 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:26.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:26.747 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:26.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:26.747 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:26Z" 2024-06-15T08:47:26.747 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:26.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.734016_560.1 2024-06-15T08:47:31.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:31.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:31.750 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:31.793 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:31.793 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:31.835 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:31.833+0000 7f6b3a7a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:31.836 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:31.837+0000 7f6b3a7a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:31.855 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:31.855 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:31.855 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:31.855 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:31.855 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:31.856 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:31.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:31.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:31.859 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:31Z" 2024-06-15T08:47:31.860 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:31.861 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:36.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:36.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:36.863 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:36.906 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:36.906 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:36.949 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:36.948+0000 7f5194a1ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:36.950 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:36.948+0000 7f5194a1ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:36.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:36.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:36.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:36.974 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:36Z" 2024-06-15T08:47:36.975 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:36.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:41.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:41.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:41.978 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:42.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:42.021 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:42.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:42.068+0000 7f9f1e9c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:42.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:42.068+0000 7f9f1e9c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:42.090 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:42.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:42.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:42.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:42.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:42Z" 2024-06-15T08:47:42.095 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:42.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:47:47.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:47.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:47.097 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:47.140 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:47.140 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:47.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:47.180+0000 7f0369a70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:47.182 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:47.180+0000 7f0369a70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:47.201 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:47Z" 2024-06-15T08:47:47.207 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:47.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:47:52.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:52.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:52.209 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:52.253 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:52.253 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:52.302 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:52.300+0000 7f64b06dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:52.303 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:52.300+0000 7f64b06dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:52.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:52.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:52.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:52.331 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:52Z" 2024-06-15T08:47:52.332 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:52.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438539.179989_590.1 2024-06-15T08:47:57.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:57.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:47:57.335 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:47:57.384 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:47:57.385 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:47:57.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:57.440+0000 7fc0af028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:57.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:47:57.440+0000 7fc0af028a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:47:57.466 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:47:57.467 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:57.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:47:57.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:57.470 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:47:57.471 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:47:57.472 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:47:57.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:47:57.473 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:47:57Z" 2024-06-15T08:47:57.473 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:47:57.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:02.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:02.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:02.475 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:02.518 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:02.518 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:02.579 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:02.576+0000 7fcea0fe1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:02.580 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:02.580+0000 7fcea0fe1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:02.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:02.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:02.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:02.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:02.610 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:02.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:02.610 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:02Z" 2024-06-15T08:48:02.610 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:02.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:07.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:07.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:07.612 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:07.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:07.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:07.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:07.708+0000 7f8341356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:07.709 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:07.708+0000 7f8341356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:07.728 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.729 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:07.730 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:07.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:07.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:07.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:07.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:07.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:07.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:07.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:07Z" 2024-06-15T08:48:07.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:07.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:12.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:12.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:12.817 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:12.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:12.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:12.909 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:12.908+0000 7f26bcb86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:12.909 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:12.908+0000 7f26bcb86a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:12.929 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:12.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:12.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:48:12.933 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:12.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:12.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:12.936 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:12.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:12.936 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:12Z" 2024-06-15T08:48:12.936 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:12.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:48:17.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:17.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:17.938 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:17.980 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:17.981 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:18.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:18.020+0000 7fb784905a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:18.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:18.020+0000 7fb784905a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:18.041 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:18.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:18.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:18.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:18.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:18.046 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:18Z" 2024-06-15T08:48:18.046 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:18.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:23.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:23.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:23.051 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:23.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:23.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:23.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:23.136+0000 7f98eea3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:23.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:23.136+0000 7f98eea3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:23.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:23.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:23Z" 2024-06-15T08:48:23.163 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:23.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:28.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:28.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:28.166 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:28.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:28.209 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:28.255 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:28.251+0000 7f211580aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:28.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:28.255+0000 7f211580aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:28.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:28.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.280 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:28.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:28.281 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:28.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:28.281 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:28Z" 2024-06-15T08:48:28.281 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:28.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:33.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:33.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:33.283 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:33.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:33.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:33.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:33.375+0000 7f72ee88aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:33.378 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:33.375+0000 7f72ee88aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:33.401 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:33Z" 2024-06-15T08:48:33.407 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:33.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:38.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:38.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:38.409 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:38.453 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:38.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:38.496 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:38.495+0000 7fe8f76bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:38.497 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:38.495+0000 7fe8f76bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:38.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:38.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:38Z" 2024-06-15T08:48:38.523 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:38.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:43.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:43.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:43.525 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:43.577 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:43.578 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:43.640 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:43.639+0000 7fd480eb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:43.641 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:43.639+0000 7fd480eb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:43.664 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:43.664 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:43.664 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:43.664 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:43.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:43.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:43.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:43.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:43Z" 2024-06-15T08:48:43.669 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:43.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:48.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:48.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:48.672 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:48.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:48.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:48.769 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:48.767+0000 7fd45993ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:48.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:48.767+0000 7fd45993ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:48.789 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:48:48.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:48.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:48Z" 2024-06-15T08:48:48.794 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:48.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:48:53.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:53.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:53.797 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:53.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:53.842 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:53.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:53.895+0000 7f184388ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:53.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:53.895+0000 7f184388ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:53.919 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:53.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:53.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:53.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:53.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:53.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:53.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:53.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:53.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:53.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:53.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:53.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:53.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:53.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:48:53.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:53.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:53.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:53.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:48:53.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:53.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:53.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:53.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:53.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:53.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:53.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:53.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:53.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:53.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:53Z" 2024-06-15T08:48:53.930 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:53.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.728887_559.1 2024-06-15T08:48:58.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:58.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:48:58.934 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:48:58.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:48:58.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:48:59.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:59.019+0000 7f964faf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:59.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:48:59.019+0000 7f964faf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:48:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:59.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:48:59.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:48:59.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:48:59Z" 2024-06-15T08:48:59.046 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:48:59.047 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:49:04.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:04.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:04.048 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:04.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:04.095 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:04.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:04.131+0000 7fcf34738a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:04.135 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:04.131+0000 7fcf34738a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:04.153 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:04.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:04.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:04.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:04.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:04.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:04.159 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:04Z" 2024-06-15T08:49:04.159 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:04.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.690655_555.1 2024-06-15T08:49:09.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:09.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:09.161 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:09.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:09.290 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:09.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:09.251+0000 7fc84eb3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:09.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:09.251+0000 7fc84eb3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.296 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:09.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:09.297 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:09.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:09.297 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:09Z" 2024-06-15T08:49:09.297 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:09.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:49:14.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:14.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:14.298 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:14.342 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:14.342 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:14.398 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:14.395+0000 7f973768fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:14.399 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:14.395+0000 7f973768fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:14.424 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:14.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:14.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:14.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:14.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:14Z" 2024-06-15T08:49:14.429 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:14.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:49:19.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:19.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:19.432 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:19.475 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:19.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:19.534 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:19.530+0000 7f1223e19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:19.535 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:19.534+0000 7f1223e19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:19.555 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:19.556 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:19.556 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:19.556 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:19.556 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:19.556 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:19.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:19.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:19.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:19.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:49:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:19Z" 2024-06-15T08:49:19.567 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:19.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:49:24.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:24.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:24.570 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:24.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:24.615 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:24.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:24.654+0000 7f973a740a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:24.657 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:24.654+0000 7f973a740a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:24.675 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:24.675 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:24.675 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:24.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:24.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:49:24.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:24.679 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:24Z" 2024-06-15T08:49:24.680 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:24.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:49:29.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:29.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:29.682 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:29.727 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:29.727 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:29.778 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:29.774+0000 7f4d21a75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:29.779 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:29.774+0000 7f4d21a75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:29.797 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:29.797 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:29.797 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.798 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:29.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:29.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:29.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:29.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:29.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:29.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:29.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:29.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:29.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:29.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.806 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:29.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:29.807 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:29.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:29.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:29Z" 2024-06-15T08:49:29.807 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:29.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441065.379479_1088.1 target=1_1718438536.699165_557.1 2024-06-15T08:49:34.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-06-15T08:49:34.811 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-67 2024-06-15T08:49:34.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:34.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:34.970 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:35.029 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:35.030 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:35.140 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:49:35.158 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:49:35.158 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:49:35.158 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:35.158 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718441374.850983_1135.1", 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:49:34.850983Z" 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:49:35.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:49:35.160 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:49:35.161 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:49:35.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:35.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:35.161 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:35.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:35.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:35.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:35.246+0000 7f82fd99aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:35.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:35.246+0000 7f82fd99aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:35.268 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:35Z" 2024-06-15T08:49:35.273 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:35.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:49:38.097 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:49:38.098+0000 7fcb789be640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:49:38.098 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:49:38.098+0000 7fcb7a1c1640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:49:40.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:40.275 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:40.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:40.386+0000 7f5fa4f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:40.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:40.386+0000 7f5fa4f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:40.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:40.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:40.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:40.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:40.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:40.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:40.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:40.414 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:40.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.414 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:40.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:40.415 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:40.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:40.415 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:40Z" 2024-06-15T08:49:40.415 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:40.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:49:45.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:45.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:45.417 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:45.462 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:45.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:45.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:45.514+0000 7f05dcb51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:45.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:45.514+0000 7f05dcb51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:45.536 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:45.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:45.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:49:45.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:45Z" 2024-06-15T08:49:45.541 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:45.542 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:49:50.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:50.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:50.544 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:50.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:50.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:50.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:50.634+0000 7fc50140ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:50.637 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:50.634+0000 7fc50140ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:50.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:50.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:50.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:50.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:50.664 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:50.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:50.664 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:50Z" 2024-06-15T08:49:50.664 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:50.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.707204_558.1 2024-06-15T08:49:55.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:55.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:49:55.666 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:49:55.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:49:55.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:49:55.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:55.758+0000 7f70310e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:55.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:49:55.766+0000 7f70310e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:49:55.788 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:49:55.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:55.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:49:55.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:49:55.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:49:55.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:49:55Z" 2024-06-15T08:49:55.794 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:49:55.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:00.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:00.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:00.796 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:00.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:00.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:00.885 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:00.886+0000 7fe1bc940a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:00.886 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:00.886+0000 7fe1bc940a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:00.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:00.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:00Z" 2024-06-15T08:50:00.910 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:00.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:05.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:05.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:05.912 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:05.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:05.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:05.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:05.998+0000 7f6fc8125a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:05.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:05.998+0000 7f6fc8125a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:06.017 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:06.018 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:06.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:06.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:06.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:06Z" 2024-06-15T08:50:06.022 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:06.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:11.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:11.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:11.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:11.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:11.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:11.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:11.113+0000 7f0a04643a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:11.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:11.113+0000 7f0a04643a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:11.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:11.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.747432_562.1", 2024-06-15T08:50:11.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.747432Z", 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:11.138 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:11.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:11.140 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:11Z" 2024-06-15T08:50:11.140 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:11.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.747432_562.1 2024-06-15T08:50:16.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:16.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:16.142 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:16.186 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:16.186 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:16.244 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:16.245+0000 7f8173df5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:16.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:16.245+0000 7f8173df5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:16.265 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:16.265 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:16.265 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:16.265 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:16.265 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:16.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:16.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.747432_562.1", 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.747432Z", 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:16.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:16.269 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:16Z" 2024-06-15T08:50:16.270 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:16.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.747432_562.1 2024-06-15T08:50:21.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:21.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:21.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:21.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:21.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:21.386 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:21.385+0000 7f98fe729a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:21.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:21.389+0000 7f98fe729a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:50:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:21.411 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:21Z" 2024-06-15T08:50:21.412 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:21.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:26.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:26.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:26.414 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:26.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:26.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:26.507 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:26.505+0000 7fb1c620da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:26.508 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:26.509+0000 7fb1c620da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:26.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:26.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:26.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:26.535 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:26.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:26.535 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:26Z" 2024-06-15T08:50:26.535 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:26.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:31.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:31.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:31.537 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:31.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:31.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:31.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:31.625+0000 7f6475b77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:31.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:31.625+0000 7f6475b77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:31.642 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:31.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:31.644 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:31.645 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:31.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:31.647 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:31Z" 2024-06-15T08:50:31.648 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:31.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:50:36.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:36.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:36.650 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:36.694 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:36.694 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:36.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:36.745+0000 7f42cb310a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:36.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:36.745+0000 7f42cb310a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:36.772 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:36.772 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:36.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:36Z" 2024-06-15T08:50:36.777 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:36.779 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:50:40.118 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:50:40.117+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:50:41.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:41.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:41.780 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:41.823 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:41.823 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:41.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:41.873+0000 7fbddefb9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:41.876 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:41.877+0000 7fbddefb9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:41.899 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:41.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:41.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:41.903 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:41.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:41.904 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:41Z" 2024-06-15T08:50:41.904 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:41.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:50:46.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:46.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:46.906 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:46.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:46.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:46.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:46.997+0000 7fb18b8eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:46.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:46.997+0000 7fb18b8eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:47.018 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:47.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:47.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:47.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:47Z" 2024-06-15T08:50:47.024 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:47.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.742181_561.1 2024-06-15T08:50:52.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:52.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:52.026 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:52.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:52.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:52.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:52.125+0000 7ff7bb7a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:52.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:52.129+0000 7ff7bb7a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:52.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:52.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:52.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:52.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:52.151 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:52.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:52.152 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:52Z" 2024-06-15T08:50:52.152 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:52.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:50:57.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:57.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:50:57.154 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:50:57.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:50:57.198 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:50:57.241 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:57.241+0000 7fa8dd5f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:57.242 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:50:57.241+0000 7fa8dd5f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:50:57.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:57.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:50:57.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:57.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:50:57.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:50:57.264 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:50:57Z" 2024-06-15T08:50:57.265 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:50:57.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:51:02.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:02.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:02.267 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:02.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:02.311 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:02.355 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:02.352+0000 7f585b012a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:02.356 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:02.356+0000 7f585b012a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:02.374 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:02.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:02.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:02.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:02.379 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:02.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:02.379 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:02Z" 2024-06-15T08:51:02.379 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:02.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:07.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:07.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:07.382 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:07.436 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:07.436 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:07.496 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:07.496+0000 7fab68192a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:07.498 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:07.496+0000 7fab68192a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:07.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:07.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:07.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:07.526 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:07Z" 2024-06-15T08:51:07.526 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:07.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:12.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:12.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:12.528 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:12.572 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:12.572 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:12.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:12.628+0000 7f6cf2e06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:12.632 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:12.632+0000 7f6cf2e06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:12.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:12.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:12.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:51:12.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:12.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:12Z" 2024-06-15T08:51:12.657 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:12.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:51:17.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:17.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:17.660 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:17.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:17.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:17.761 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:17.760+0000 7f6e5bf9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:17.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:17.760+0000 7f6e5bf9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:17.785 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:17.785 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:17.785 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:17.785 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:17.786 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:17.786 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:17.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:17.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.786 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:17.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:17.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:17.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:17.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:17Z" 2024-06-15T08:51:17.792 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:17.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:51:22.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:22.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:22.794 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:22.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:22.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:22.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:22.880+0000 7f7666be6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:22.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:22.880+0000 7f7666be6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:22.900 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:22.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:22.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:22.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:22.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:22Z" 2024-06-15T08:51:22.905 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:22.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.742181_561.1 2024-06-15T08:51:27.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:27.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:27.960 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:28.004 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:28.004 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:28.068 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:28.068+0000 7f4589e2ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:28.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:28.068+0000 7f4589e2ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:28Z" 2024-06-15T08:51:28.098 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:28.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:33.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:33.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:33.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:33.160 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:33.161 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:33.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:33.208+0000 7f9cf5a60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:33.210 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:33.208+0000 7f9cf5a60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:33.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:33.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:33.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:33.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:33Z" 2024-06-15T08:51:33.239 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:33.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:38.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:38.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:38.241 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:38.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:38.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:38.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:38.344+0000 7f3a890dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:38.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:38.344+0000 7f3a890dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:38.365 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:38.365 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:38.365 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:38.365 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:38.365 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:38.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:38.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:38.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:38.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:38.370 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:38Z" 2024-06-15T08:51:38.371 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:38.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:43.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:43.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:43.373 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:43.418 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:43.418 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:43.462 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:43.460+0000 7effe39caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:43.463 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:43.460+0000 7effe39caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:43.483 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:43.483 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:43.483 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:43.483 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:43.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:43.485 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:43.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:43.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:43Z" 2024-06-15T08:51:43.488 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:43.489 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:48.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:48.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:48.491 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:48.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:48.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:48.591+0000 7f526e59aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:48.595+0000 7f526e59aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:48.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:48.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:48.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:48.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:48.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:48.637 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:48Z" 2024-06-15T08:51:48.637 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:48.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:51:53.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:53.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:53.639 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:53.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:53.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:53.739 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:53.735+0000 7f128005fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:53.740 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:53.739+0000 7f128005fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:53.767 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:53.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:53.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:53.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:53.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:53.773 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:53Z" 2024-06-15T08:51:53.774 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:53.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:51:58.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:58.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:51:58.775 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:51:58.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:51:58.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:51:58.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:58.863+0000 7fc15100ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:58.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:51:58.863+0000 7fc15100ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:51:58.890 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.923 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:58.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:51:58.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:51:58.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:51:58.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:51:58.928 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:51:58Z" 2024-06-15T08:51:58.928 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:51:58.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.707204_558.1 2024-06-15T08:52:03.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:03.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:03.930 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:03.973 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:03.973 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:04.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:04.015+0000 7f27b4ea9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:04.017 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:04.015+0000 7f27b4ea9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:04.035 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:04.035 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:04.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:04.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:04.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:04.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:04.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:04.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:04.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:04.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:04.041 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:04Z" 2024-06-15T08:52:04.041 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:04.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:09.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:09.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:09.043 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:09.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:09.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:09.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:09.131+0000 7f71d2f09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:09.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:09.135+0000 7f71d2f09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:09.157 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:09.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:09.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:09.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:09.161 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:09Z" 2024-06-15T08:52:09.162 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:09.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:14.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:14.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:14.164 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:14.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:14.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:14.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:14.255+0000 7f5a4240da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:14.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:14.259+0000 7f5a4240da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:14.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:14.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:14.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:14.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:14.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:14.291 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:14.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:14.291 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:14Z" 2024-06-15T08:52:14.291 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:14.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.707204_558.1 2024-06-15T08:52:19.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:19.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:19.293 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:19.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:19.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:19.392 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:19.391+0000 7f534f58ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:19.393 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:19.391+0000 7f534f58ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.420 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:19.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:19.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:19.424 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:19.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:19.425 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:19Z" 2024-06-15T08:52:19.425 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:19.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:52:24.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:24.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:24.427 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:24.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:24.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:24.532 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:24.531+0000 7f7449326a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:24.533 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:24.531+0000 7f7449326a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:24.554 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:24.554 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:24.555 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:24.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:24.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:24Z" 2024-06-15T08:52:24.559 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:24.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:29.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:29.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:29.562 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:29.600 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:29.652 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:29.660 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:29.659+0000 7fee76b1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:29.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:29.659+0000 7fee76b1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:29.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:29.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:29.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:29.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:29.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:29Z" 2024-06-15T08:52:29.688 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:29.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:34.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:34.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:34.691 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:34.735 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:34.735 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:34.779 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:34.774+0000 7fea8ccdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:34.780 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:34.778+0000 7fea8ccdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:34.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:34.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:34.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:34.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:34.802 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:34Z" 2024-06-15T08:52:34.803 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:34.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:39.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:39.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:39.805 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:39.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:39.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:39.916 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:39.914+0000 7f0e2e1fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:39.917 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:39.914+0000 7f0e2e1fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:39.942 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:39.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:39.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.946 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:39.947 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:39Z" 2024-06-15T08:52:39.947 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:39.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:44.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:44.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:44.949 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:44.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:44.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:45.033 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:45.030+0000 7fce3f23fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:45.034 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:45.030+0000 7fce3f23fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.053 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:45.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.055 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:45.056 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:45.057 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:45.058 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:45.058 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:45.058 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:45Z" 2024-06-15T08:52:45.058 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:45.059 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:52:50.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:50.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:50.060 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:50.103 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:50.104 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:50.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:50.150+0000 7f44cd62ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:50.154 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:50.150+0000 7f44cd62ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:50.175 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:50.175 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:50.175 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:50.175 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:50.176 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:50.177 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:50.178 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:50.179 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:50.180 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:50Z" 2024-06-15T08:52:50.181 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:50.182 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:52:55.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:55.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:52:55.184 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:52:55.227 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:52:55.282 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:52:55.283 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:55.278+0000 7f11bd5e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:55.283 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:52:55.278+0000 7f11bd5e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:52:55.301 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:52:55.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:52:55.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:55.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:55.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:52:55.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:52:55.307 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:52:55.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:52:55.308 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:52:55Z" 2024-06-15T08:52:55.308 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:52:55.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:53:00.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:00.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:00.311 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:00.354 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:00.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:00.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:00.410+0000 7fe57f89ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:00.416 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:00.410+0000 7fe57f89ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:00.438 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:00.439 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:00.439 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:00.439 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:00.439 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:00.440 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:00.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:00.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.440 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:00.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:00.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:00.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:00.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:00.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:00.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:00.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:00.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:00.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:00.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:00.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:00.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:00.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:00.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:00.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:00.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:00.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:53:00.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:00.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:00.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:00.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:53:00.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:00.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:00.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:00.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:00.449 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:00.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.449 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:00.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:00.450 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:00.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:00.450 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:00Z" 2024-06-15T08:53:00.450 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:00.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.707204_558.1 2024-06-15T08:53:05.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:05.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:05.453 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:05.497 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:05.497 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:05.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:05.538+0000 7f1892766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:05.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:05.538+0000 7f1892766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.565 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:05.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:05.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:05.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:05.570 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:05.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:05.570 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:05Z" 2024-06-15T08:53:05.570 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:05.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:53:10.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:10.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:10.572 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:10.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:10.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:10.659 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:10.654+0000 7f72690eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:10.660 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:10.658+0000 7f72690eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:10.678 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:10.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.681 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:10.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:10.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:10Z" 2024-06-15T08:53:10.684 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:10.686 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:53:15.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:15.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:15.687 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:15.730 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:15.731 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:15.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:15.774+0000 7fc86e6d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:15.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:15.774+0000 7fc86e6d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:15.792 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:15.792 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:15.792 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:15.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:15.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:15.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:15.796 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:15Z" 2024-06-15T08:53:15.797 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:15.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:53:20.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:20.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:20.800 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:20.844 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:20.844 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:20.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:20.898+0000 7efe587d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:20.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:20.898+0000 7efe587d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:20.916 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:20.916 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:20.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:20.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:20.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:20.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:20.921 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:20Z" 2024-06-15T08:53:20.922 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:20.923 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:53:25.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:25.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:25.924 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:25.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:25.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:26.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:26.021+0000 7fdd56ed4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:26.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:26.025+0000 7fdd56ed4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:26.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:26.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:26.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:53:26.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:26.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:26Z" 2024-06-15T08:53:26.051 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:26.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:53:31.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:31.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:31.054 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:31.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:31.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:31.158 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:31.157+0000 7f43c8482a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:31.159 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:31.157+0000 7f43c8482a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:31.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:31.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:31.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:31.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:31.185 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:31.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:31.185 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:31Z" 2024-06-15T08:53:31.185 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:31.186 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.979596_577.1 2024-06-15T08:53:36.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:36.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:36.187 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:36.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:36.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:36.290 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:36.289+0000 7f46b164aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:36.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:36.289+0000 7f46b164aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:36.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:36.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:36.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:36.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:36.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:36.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:36Z" 2024-06-15T08:53:36.318 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:36.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:53:41.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:41.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:41.320 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:41.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:41.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:41.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:41.405+0000 7f81a527da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:41.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:41.405+0000 7f81a527da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:41.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:41.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:41.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.430 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:41.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:41.431 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:41.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:41.431 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:41Z" 2024-06-15T08:53:41.431 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:41.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:53:46.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:46.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:46.433 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:46.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:46.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:46.530 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:46.529+0000 7f9930767a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:46.531 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:46.529+0000 7f9930767a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:46.552 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:46.552 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:46.552 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:46.552 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:46.552 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:46.554 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:46.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:46.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:46.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:46.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:46Z" 2024-06-15T08:53:46.560 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:46.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:53:51.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:51.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:51.563 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:51.607 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:51.607 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:51.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:51.653+0000 7f828d89aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:51.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:51.653+0000 7f828d89aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:51.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:51.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:51.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:51.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:51Z" 2024-06-15T08:53:51.679 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:51.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:53:56.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:56.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:53:56.682 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:53:56.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:53:56.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:53:56.757 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:56.757+0000 7ffb0f7dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:56.758 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:53:56.757+0000 7ffb0f7dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:53:56.776 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:56.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:53:56.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:56.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:53:56.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:53:56.781 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:53:56Z" 2024-06-15T08:53:56.782 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:53:56.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:54:01.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:01.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:01.784 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:01.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:01.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:01.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:01.873+0000 7f52bd1c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:01.873 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:01.873+0000 7f52bd1c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:01.895 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:01Z" 2024-06-15T08:54:01.901 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:01.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.728887_559.1 2024-06-15T08:54:06.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:06.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:06.904 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:06.946 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:06.946 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:06.996 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:06.993+0000 7f646dda3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:06.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:06.997+0000 7f646dda3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:07.017 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:07.017 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:07.017 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:07.018 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:07.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:07.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:07.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:07Z" 2024-06-15T08:54:07.022 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:07.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.699165_557.1 2024-06-15T08:54:12.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:12.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:12.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:12.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:12.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:12.111 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:12.108+0000 7fd59297ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:12.111 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:12.112+0000 7fd59297ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:12.130 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:12.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:12.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:12.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:12.134 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:12Z" 2024-06-15T08:54:12.135 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:12.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:54:17.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:17.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:17.138 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:17.181 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:17.181 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:17.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:17.220+0000 7f8b4ae66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:17.224 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:17.224+0000 7f8b4ae66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.245 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:17.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:17.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:17.249 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:17.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:17.250 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:17Z" 2024-06-15T08:54:17.250 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:17.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:54:22.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:22.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:22.252 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:22.295 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:22.296 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:22.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:22.336+0000 7f5a05912a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:22.341 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:22.340+0000 7f5a05912a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:22.359 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:22.359 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:22.360 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:22.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:22.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:22Z" 2024-06-15T08:54:22.364 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:22.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:54:27.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:27.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:27.367 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:27.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:27.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:27.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:27.460+0000 7f7001adaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:27.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:27.460+0000 7f7001adaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:27.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:27.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:27.485 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:27.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:27Z" 2024-06-15T08:54:27.487 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:27.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.690655_555.1 2024-06-15T08:54:32.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:32.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:32.490 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:32.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:32.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:32.601 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:32.600+0000 7fae948cda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:32.602 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:32.600+0000 7fae948cda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:32.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:32.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:32.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:32.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:32.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:32Z" 2024-06-15T08:54:32.628 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:32.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.699165_557.1 2024-06-15T08:54:37.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:37.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:37.631 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:37.674 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:37.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:37.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:37.716+0000 7f1faed7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:37.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:37.720+0000 7f1faed7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:37.738 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:37.738 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:37Z" 2024-06-15T08:54:37.743 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:37.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441374.850983_1135.1 target=1_1718438536.699165_557.1 2024-06-15T08:54:42.746 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... FAIL 2024-06-15T08:54:42.746 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-68 2024-06-15T08:54:42.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:42.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:42.962 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:43.025 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:43.141 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:54:43.156 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:54:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:54:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:54:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718441682.790801_1162.1", 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:54:42.790801Z" 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:54:43.157 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:54:43.158 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:54:43.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:43.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:43.159 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:43.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:43.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:43.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:43.252+0000 7f39bc044a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:43.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:43.252+0000 7f39bc044a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:43.272 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:43.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:43.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:43.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:43Z" 2024-06-15T08:54:43.277 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:43.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:54:47.954 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:54:47.951+0000 7fcb709ae640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T08:54:48.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:48.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:48.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:48.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:48.325 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:48.370 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:48.368+0000 7f4ae5895a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:48.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:48.368+0000 7f4ae5895a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:48.389 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:48.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:48.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:48.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:48.393 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:48Z" 2024-06-15T08:54:48.394 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:48.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:54:53.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:53.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:53.397 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:53.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:53.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:53.484 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:53.484+0000 7f32c5156a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:53.485 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:53.484+0000 7f32c5156a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:53.504 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:53.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:53.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:53.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:53.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:53Z" 2024-06-15T08:54:53.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:53.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:54:58.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:58.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:54:58.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:54:58.555 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:54:58.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:54:58.610 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:58.608+0000 7f77550f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:58.611 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:54:58.608+0000 7f77550f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:54:58.631 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:54:58.632 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:58.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:58.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:54:58.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:54:58.636 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:54:58.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:54:58.637 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:54:58Z" 2024-06-15T08:54:58.637 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:54:58.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:55:03.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:03.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:03.640 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:03.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:03.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:03.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:03.739+0000 7ff9ed9afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:03.744 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:03.743+0000 7ff9ed9afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:03.769 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:03.769 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:03.769 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:03.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:03.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:03.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:03.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:03.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.775 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:03.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:03.776 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:03.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:03.776 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:03Z" 2024-06-15T08:55:03.776 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:03.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:55:08.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:08.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:08.778 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:08.821 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:08.822 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:08.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:08.871+0000 7f2a00ce5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:08.876 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:08.875+0000 7f2a00ce5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:08.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:08.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:08.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:08.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:08.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:08.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:08.903 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:08Z" 2024-06-15T08:55:08.903 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:08.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:55:13.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:13.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:13.905 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:13.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:13.951 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:13.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:13.991+0000 7fb2e8129a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:13.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:13.991+0000 7fb2e8129a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:14.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:14.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:14.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:14.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:14.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:14.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:55:14.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:14.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:13Z" 2024-06-15T08:55:14.016 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:14.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:55:19.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:19.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:19.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:19.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:19.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:19.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:19.119+0000 7f1698ff5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:19.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:19.119+0000 7f1698ff5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:19.144 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:19.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:19.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:19.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:19.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:19.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:19.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:19.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:19.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:19.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:19.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:19.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:19.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:19.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:19.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:55:19.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:19.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:19.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:19.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:19.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:19.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:19.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:19.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:19.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:19.156 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:19Z" 2024-06-15T08:55:19.156 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:19.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:55:24.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:24.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:24.159 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:24.203 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:24.203 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:24.266 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:24.263+0000 7fe3e9539a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:24.267 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:24.263+0000 7fe3e9539a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:24.289 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:24.289 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:24.289 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:24.289 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:24.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:24.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:24.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:24Z" 2024-06-15T08:55:24.294 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:24.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:55:29.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:29.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:29.296 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:29.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:29.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:29.381 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:29.379+0000 7fd0ba685a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:29.382 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:29.379+0000 7fd0ba685a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:29.403 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:29.403 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:29.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:29.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:55:29.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:29.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:29Z" 2024-06-15T08:55:29.408 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:29.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:55:34.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:34.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:34.411 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:34.453 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:34.454 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:34.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:34.499+0000 7f8c3d1f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:34.502 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:34.499+0000 7f8c3d1f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:34.523 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:34.523 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:34.523 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:34.523 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:34.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:34.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:34.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.528 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:34.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:34.529 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:34.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:34.529 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:34Z" 2024-06-15T08:55:34.529 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:34.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.955581_573.1 2024-06-15T08:55:39.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:39.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:39.531 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:39.614 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:39.611+0000 7f77fe27ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:39.615 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:39.611+0000 7f77fe27ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:39.633 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:39Z" 2024-06-15T08:55:39.638 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:39.639 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:55:40.138 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:55:40.135+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T08:55:44.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:44.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:44.640 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:44.684 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:44.684 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:44.731 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:44.727+0000 7f01529e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:44.733 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:44.731+0000 7f01529e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:44.753 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:44.754 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:44.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:44.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:44.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:44Z" 2024-06-15T08:55:44.758 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:44.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:55:49.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:49.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:49.760 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:49.818 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:49.818 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:49.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:49.863+0000 7f8470109a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:49.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:49.863+0000 7f8470109a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:49.889 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:49.889 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:49.889 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:49.889 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:49.889 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:49.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:49.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:49.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:49.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:49Z" 2024-06-15T08:55:49.894 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:49.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.615034_551.1 2024-06-15T08:55:54.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:54.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:55:54.899 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:55:54.942 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:55:54.942 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:55:54.983 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:54.978+0000 7fd3f01bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:54.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:55:54.982+0000 7fd3f01bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.005 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.006 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:55:55.007 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:55:55.008 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:55:54Z" 2024-06-15T08:55:55.009 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:55:55.010 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:56:00.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:00.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:00.012 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:00.055 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:00.055 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:00.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:00.130+0000 7f25c4838a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:00.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:00.130+0000 7f25c4838a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:00.170 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:00.170 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:00.170 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:00.170 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:00.170 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:00.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:00.172 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:56:00.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:00.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:00Z" 2024-06-15T08:56:00.175 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:00.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.699165_557.1 2024-06-15T08:56:05.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:05.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:05.178 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:05.221 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:05.222 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:05.262 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:05.258+0000 7fc3c6da8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:05.263 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:05.258+0000 7fc3c6da8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:05.283 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:05.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:05.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.287 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:05.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:05.288 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:05.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:05.288 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:05Z" 2024-06-15T08:56:05.288 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:05.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:56:10.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:10.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:10.291 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:10.336 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:10.336 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:10.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:10.374+0000 7f6f13c60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:10.378 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:10.374+0000 7f6f13c60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:10.400 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:10.400 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:10.400 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:10.400 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:10.400 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:10.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:10.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:10.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:10.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:10.405 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:10.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:10.405 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:10Z" 2024-06-15T08:56:10.405 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:10.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:56:15.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:15.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:15.407 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:15.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:15.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:15.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:15.506+0000 7fcbb2244a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:15.510 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:15.506+0000 7fcbb2244a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:15.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:15.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:15.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:15Z" 2024-06-15T08:56:15.562 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:15.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:56:20.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:20.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:20.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:20.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:20.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:20.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:20.662+0000 7f7478c50a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:20.668 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:20.666+0000 7f7478c50a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:20.694 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:20.694 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:20.694 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.695 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:20.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:20.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:20.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:20.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:20.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:20.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:20.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:20.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:20.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:20.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:20.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:20.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.703 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:20.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:20.704 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:20.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:20.704 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:20Z" 2024-06-15T08:56:20.704 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:20.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:56:25.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:25.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:25.707 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:25.750 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:25.750 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:25.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:25.806+0000 7f977a9fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:25.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:25.806+0000 7f977a9fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:25.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:25.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:25.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:25.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:25.841 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:25.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:25.841 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:25Z" 2024-06-15T08:56:25.841 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:25.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.959215_574.1 2024-06-15T08:56:30.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:30.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:30.843 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:30.884 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:30.884 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:30.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:30.922+0000 7f66c77c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:30.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:30.926+0000 7f66c77c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:30.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:30.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:30.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:30.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:30.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:30Z" 2024-06-15T08:56:30.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:30.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:56:35.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:35.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:35.956 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:36.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:36.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:36.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:36.046+0000 7fb66c8d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:36.047 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:36.046+0000 7fb66c8d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:36.065 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:36.065 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:36.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:36.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:56:36.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:36.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:36.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:36.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:36.071 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:36Z" 2024-06-15T08:56:36.072 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:36.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:56:41.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:41.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:41.074 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:41.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:41.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:41.159 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:41.158+0000 7ff0aff3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:41.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:41.162+0000 7ff0aff3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:41.178 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:41.178 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:41.178 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:41.178 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:41.178 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:41.179 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:41.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:41.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:41.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:41Z" 2024-06-15T08:56:41.183 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:41.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:56:46.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:46.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:46.185 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:46.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:46.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:46.275 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:46.273+0000 7fb613d1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:46.276 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:46.277+0000 7fb613d1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:46.294 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:46.294 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:46.294 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:46.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:46.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:46.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:46.298 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:46.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:46.299 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:46Z" 2024-06-15T08:56:46.299 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:46.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:56:51.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:51.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:51.301 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:51.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:51.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:51.395 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:51.393+0000 7fc3b8307a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:51.396 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:51.397+0000 7fc3b8307a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:51.414 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:51.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:51.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:51.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:51.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:51Z" 2024-06-15T08:56:51.419 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:51.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:56:56.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:56.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:56:56.421 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:56:56.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:56:56.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:56:56.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:56.521+0000 7fee1b970a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:56.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:56:56.521+0000 7fee1b970a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:56.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:56.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:56:56.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:56:56.548 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:56:56.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:56:56.549 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:56:56Z" 2024-06-15T08:56:56.549 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:56:56.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:01.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:01.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:01.551 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:01.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:01.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:01.635 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:01.633+0000 7fedf44cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:01.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:01.637+0000 7fedf44cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:01.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:01.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T08:57:01.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:01.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:01Z" 2024-06-15T08:57:01.659 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:01.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.895427_569.1 2024-06-15T08:57:06.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:06.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:06.662 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:06.712 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:06.712 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:06.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:06.765+0000 7fb7dcc6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:06.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:06.765+0000 7fb7dcc6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:06.787 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:06.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:06.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:06.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:06Z" 2024-06-15T08:57:06.792 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:06.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:57:11.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:11.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:11.795 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:11.838 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:11.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:11.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:11.901+0000 7f7a29168a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:11.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:11.901+0000 7f7a29168a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:11.928 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:11.928 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:11.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:11.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:11.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:11.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:11Z" 2024-06-15T08:57:11.933 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:11.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:16.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:16.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:16.936 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:16.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:16.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:17.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:17.025+0000 7f0f8fa0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:17.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:17.025+0000 7f0f8fa0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:17.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:17.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:17.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:17.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:17.051 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:17Z" 2024-06-15T08:57:17.052 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:17.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:22.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:22.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:22.054 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:22.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:22.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:22.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:22.141+0000 7f0a39957a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:22.144 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:22.145+0000 7f0a39957a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:22.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:22.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:22.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.167 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:22.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:22.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:22.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:22.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:22Z" 2024-06-15T08:57:22.168 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:22.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.742181_561.1 2024-06-15T08:57:27.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:27.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:27.170 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:27.215 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:27.215 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:27.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:27.257+0000 7f473ac78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:27.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:27.257+0000 7f473ac78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:27.278 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:27.278 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:27.278 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:27.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:27.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:27.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:27.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:27Z" 2024-06-15T08:57:27.283 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:27.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.742181_561.1 2024-06-15T08:57:32.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:32.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:32.285 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:32.330 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:32.331 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:32.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:32.368+0000 7f8b23fb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:32.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:32.372+0000 7f8b23fb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:32.405 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:32.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:32.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:32.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:32.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:32.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:32.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:32.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.412 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:32.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:32.413 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:32.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:32.413 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:32Z" 2024-06-15T08:57:32.413 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:32.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:37.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:37.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:37.416 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:37.460 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:37.460 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:37.519 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:37.516+0000 7f6c4b632a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:37.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:37.520+0000 7f6c4b632a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:37.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:37.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:37.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:37.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:37.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:37Z" 2024-06-15T08:57:37.551 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:37.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:57:42.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:42.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:42.553 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:42.596 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:42.596 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:42.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:42.652+0000 7f57a5478a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:42.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:42.652+0000 7f57a5478a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:42.673 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:42.673 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:42.673 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:42.673 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:42.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:42.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:42.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:42.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:42Z" 2024-06-15T08:57:42.678 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:42.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:47.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:47.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:47.681 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:47.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:47.724 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:47.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:47.764+0000 7fd41cdc6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:47.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:47.768+0000 7fd41cdc6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:47.788 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:47.788 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:47.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:47.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:47.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:47.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:47Z" 2024-06-15T08:57:47.793 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:47.794 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:52.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:52.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:52.796 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:52.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:52.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:52.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:52.892+0000 7fee7556da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:52.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:52.896+0000 7fee7556da40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:52.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:52.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:52.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:52.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:52.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:52.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:52.920 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:52Z" 2024-06-15T08:57:52.920 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:52.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:57:57.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:57.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:57:57.922 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:57:57.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:57:57.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:57:58.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:58.012+0000 7f26b5acfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:58.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:57:58.012+0000 7f26b5acfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:57:58.034 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:58.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:58.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:58.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:57:58.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:57:58Z" 2024-06-15T08:57:58.039 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:57:58.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.699165_557.1 2024-06-15T08:58:03.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:03.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:03.042 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:03.084 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:03.085 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:03.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:03.136+0000 7f2d8bfeda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:03.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:03.136+0000 7f2d8bfeda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.157 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:03.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:03.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:03.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:03.161 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:03Z" 2024-06-15T08:58:03.162 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:03.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:58:08.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:08.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:08.164 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:08.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:08.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:08.277 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:08.276+0000 7fb2eb231a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:08.279 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:08.276+0000 7fb2eb231a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:08.306 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:08.306 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:08.306 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:08.306 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:08.306 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:08.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:08.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:08.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:08.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:08.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:08.312 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:08Z" 2024-06-15T08:58:08.312 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:08.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:58:13.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:13.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:13.315 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:13.357 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:13.357 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:13.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:13.400+0000 7f65610d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:13.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:13.400+0000 7f65610d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:13.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:13.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.160495_588.1", 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.160495Z", 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:13.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:13Z" 2024-06-15T08:58:13.429 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:13.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438539.160495_588.1 2024-06-15T08:58:18.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:18.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:18.431 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:18.472 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:18.472 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:18.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:18.520+0000 7fe733813a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:18.524 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:18.524+0000 7fe733813a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:18.544 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:18.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:18.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:18.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:18.549 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:18.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:18.549 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:18Z" 2024-06-15T08:58:18.549 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:18.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:58:23.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:23.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:23.551 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:23.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:23.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:23.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:23.643+0000 7f2d2e83aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:23.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:23.647+0000 7f2d2e83aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:23.669 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:23.669 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:23.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:23.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:23.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:23.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:23.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:23.675 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:23.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:23.675 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:23Z" 2024-06-15T08:58:23.675 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:23.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:58:28.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:28.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:28.677 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:28.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:28.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:28.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:28.763+0000 7fa9b5054a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:28.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:28.763+0000 7fa9b5054a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:28.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:28.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:28.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:28.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:28Z" 2024-06-15T08:58:28.788 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:28.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:58:33.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:33.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:33.791 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:33.834 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:33.834 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:33.903 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:33.899+0000 7fa9780eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:33.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:33.899+0000 7fa9780eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:33.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:33.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:33.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:33.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.928 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:33.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:33.929 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:33.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:33.929 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:33Z" 2024-06-15T08:58:33.929 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:33.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:58:38.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:38.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:38.931 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:38.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:38.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:39.029 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:39.027+0000 7f448ba96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:39.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:39.027+0000 7f448ba96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.051 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:39.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:39.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:39.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:39.056 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:39.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:39.056 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:39Z" 2024-06-15T08:58:39.056 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:39.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:58:44.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:44.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:44.058 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:44.101 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:44.101 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:44.152 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:44.147+0000 7f726408aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:44.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:44.151+0000 7f726408aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:44.172 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:44.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:44.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:44.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:44.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:44Z" 2024-06-15T08:58:44.177 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:44.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:58:49.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:49.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:49.179 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:49.222 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:49.222 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:49.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:49.279+0000 7f616082ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:49.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:49.279+0000 7f616082ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:49.315 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:49.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:49.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:49.320 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:49Z" 2024-06-15T08:58:49.321 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:49.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:58:54.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:54.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:54.323 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:54.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:54.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:54.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:54.411+0000 7f28522efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:54.416 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:54.415+0000 7f28522efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:54.442 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:54.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:54.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:54Z" 2024-06-15T08:58:54.447 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:54.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.699165_557.1 2024-06-15T08:58:59.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:59.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:58:59.449 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:58:59.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:58:59.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:58:59.561 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:59.559+0000 7fbdb7f9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:59.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:58:59.559+0000 7fbdb7f9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.586 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:58:59.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:58:59.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:58:59.590 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:58:59.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:58:59.591 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:58:59Z" 2024-06-15T08:58:59.591 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:58:59.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.707204_558.1 2024-06-15T08:59:04.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:04.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:04.593 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:04.636 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:04.637 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:04.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:04.683+0000 7fa7cfb24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:04.686 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:04.683+0000 7fa7cfb24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:04.706 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:04.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:04.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:04Z" 2024-06-15T08:59:04.709 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:04.710 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:09.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:09.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:09.711 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:09.754 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:09.754 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:09.804 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:09.803+0000 7f0d17405a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:09.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:09.803+0000 7f0d17405a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:09.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:09.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:09Z" 2024-06-15T08:59:09.836 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:09.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:14.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:14.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:14.839 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:14.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:14.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:14.949 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:14.946+0000 7f135db96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:14.950 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:14.946+0000 7f135db96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:14.980 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:14.980 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:14.980 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:14.980 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:14.980 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:14.981 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:14.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:14.983 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:14.984 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:14.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:14.985 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:14Z" 2024-06-15T08:59:14.985 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:14.986 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:59:19.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:19.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:19.987 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:20.029 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:20.030 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:20.085 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:20.082+0000 7f17e0f4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:20.086 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:20.082+0000 7f17e0f4aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:20.110 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:20.111 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:20.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:59:20.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:20.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:20Z" 2024-06-15T08:59:20.115 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:20.116 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:59:25.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:25.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:25.118 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:25.159 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:25.159 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:25.222 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:25.218+0000 7f8dc8474a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:25.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:25.218+0000 7f8dc8474a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:25.243 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:25.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:25.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:25.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:25.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:25Z" 2024-06-15T08:59:25.248 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:25.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:30.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:30.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:30.251 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:30.294 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:30.294 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:30.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:30.338+0000 7f273d379a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:30.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:30.338+0000 7f273d379a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:30.367 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:30.367 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:30.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:30Z" 2024-06-15T08:59:30.372 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:30.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:35.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:35.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:35.375 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:35.421 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:35.421 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:35.467 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:35.462+0000 7f777cd79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:35.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:35.462+0000 7f777cd79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:35.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:35.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:35.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:35.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:35.549 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:35Z" 2024-06-15T08:59:35.549 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:35.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:40.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:40.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:40.551 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:40.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:40.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:40.637 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:40.634+0000 7f913ad08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:40.638 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:40.634+0000 7f913ad08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:40.657 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:40.657 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:40.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:40.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:40.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:40.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:40.662 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:40.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:40.662 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:40Z" 2024-06-15T08:59:40.662 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:40.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.728887_559.1 2024-06-15T08:59:45.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:45.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:45.664 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:45.707 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:45.707 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:45.771 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:45.766+0000 7f9d2c564a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:45.772 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:45.766+0000 7f9d2c564a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:45.796 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:45.796 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:45.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:45.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:59:45.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:45.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:45Z" 2024-06-15T08:59:45.801 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:45.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441682.790801_1162.1 target=1_1718438536.690655_555.1 2024-06-15T08:59:50.804 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-06-15T08:59:50.804 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xdvrxv-69 2024-06-15T08:59:51.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:51.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:51.073 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:51.111 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:51.111 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:51.173 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T08:59:51.201 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718441990.938224_1201.1", 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T08:59:50.938224Z" 2024-06-15T08:59:51.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T08:59:51.203 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T08:59:51.203 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:51.203 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T08:59:51.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T08:59:51.203 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T08:59:51.204 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T08:59:51.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:51.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:51.204 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:51.247 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:51.247 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:51.287 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:51.282+0000 7f343ea10a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:51.288 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:51.286+0000 7f343ea10a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:51.307 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:51.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:51.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:59:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:51Z" 2024-06-15T08:59:51.312 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:51.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T08:59:56.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:56.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T08:59:56.315 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T08:59:56.360 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T08:59:56.360 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T08:59:56.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:56.418+0000 7fa018ec4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:56.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T08:59:56.418+0000 7fa018ec4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T08:59:56.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T08:59:56.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T08:59:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T08:59:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T08:59:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T08:59:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T08:59:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T08:59:56Z" 2024-06-15T08:59:56.442 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T08:59:56.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T08:59:57.869 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T08:59:57.865+0000 7fcb709ae640 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-06-15T09:00:01.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:01.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:01.445 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:01.488 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:01.488 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:01.531 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:01.530+0000 7f5d4098ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:01.532 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:01.530+0000 7f5d4098ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:01.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:01.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:01.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:01.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:01.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:01.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:01Z" 2024-06-15T09:00:01.563 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:01.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.699165_557.1 2024-06-15T09:00:06.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:06.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:06.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:06.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:06.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:06.652 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:06.653+0000 7f7975b2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:06.653 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:06.653+0000 7f7975b2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:06.671 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:06.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:06.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:06.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:06.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:06.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:06.677 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:06.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:06.677 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:06Z" 2024-06-15T09:00:06.677 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:06.678 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:00:11.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:11.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:11.680 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:11.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:11.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:11.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:11.773+0000 7fdde52b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:11.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:11.773+0000 7fdde52b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:11.800 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:11.800 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:11.800 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:11.800 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:11.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:11.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:11.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:11.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:11.802 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:11.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:11.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:11.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:11.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:11.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:11.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:11.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:11.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:11.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:11.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:11.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:11.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:11.811 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:11Z" 2024-06-15T09:00:11.811 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:11.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:16.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:16.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:16.814 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:16.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:16.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:16.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:16.901+0000 7f934dc07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:16.903 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:16.901+0000 7f934dc07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:16.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.983133_578.1", 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.983133Z", 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:16Z" 2024-06-15T09:00:16.926 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:16.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.983133_578.1 2024-06-15T09:00:21.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:21.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:21.928 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:21.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:21.972 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:22.027 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:22.025+0000 7f5bd3043a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:22.028 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:22.025+0000 7f5bd3043a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:22.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:22.054 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:22Z" 2024-06-15T09:00:22.055 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:22.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:27.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:27.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:27.057 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:27.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:27.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:27.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:27.141+0000 7f43ab047a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:27.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:27.141+0000 7f43ab047a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:27.161 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:27.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:27.166 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:27.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:27.167 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:27Z" 2024-06-15T09:00:27.167 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:27.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:32.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:32.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:32.169 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:32.211 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:32.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:32.272 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:32.269+0000 7f90890dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:32.273 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:32.273+0000 7f90890dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:32.302 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:32.302 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:32.302 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:32.302 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:32.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:32.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:32.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:32.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:32.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:32.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:32.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:32.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:32.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:32.309 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:32.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:32.310 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:32Z" 2024-06-15T09:00:32.310 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:32.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.707204_558.1 2024-06-15T09:00:37.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:37.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:37.312 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:37.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:37.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:37.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:37.405+0000 7f6d47844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:37.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:37.405+0000 7f6d47844a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:37.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:37.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:37Z" 2024-06-15T09:00:37.433 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:37.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:42.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:42.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:42.435 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:42.479 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:42.479 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:42.535 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:42.533+0000 7f150315ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:42.536 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:42.533+0000 7f150315ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:42.555 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:42.561 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:42Z" 2024-06-15T09:00:42.562 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:42.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:47.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:47.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:47.564 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:47.608 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:47.608 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:47.660 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:47.661+0000 7f06b8eb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:47.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:47.661+0000 7f06b8eb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:47.681 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:47.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:47.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:47Z" 2024-06-15T09:00:47.686 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:47.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:00:52.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:52.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:52.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:52.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:52.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:52.793 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:52.793+0000 7f3329e46a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:52.795 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:52.793+0000 7f3329e46a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:52.820 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:52.820 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:52.820 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:52.820 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:52.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:52.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:52.823 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:52.824 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:52Z" 2024-06-15T09:00:52.825 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:52.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:00:57.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:57.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:00:57.828 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:00:57.855 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T09:00:57.852+0000 7fcb6699a640 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-06-15T09:00:57.873 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:00:57.873 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:00:57.922 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:57.920+0000 7f9287455a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:57.923 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:00:57.920+0000 7f9287455a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:00:57.945 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:57.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:00:57.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.615034_551.1", 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.615034Z", 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:57.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:00:57.949 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:00:57Z" 2024-06-15T09:00:57.950 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:00:57.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.615034_551.1 2024-06-15T09:01:02.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:02.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:02.953 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:03.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:03.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:03.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:03.072+0000 7fcb45d51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:03.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:03.072+0000 7fcb45d51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:03.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:03.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:03.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:03.102 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:03Z" 2024-06-15T09:01:03.103 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:03.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:01:08.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:08.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:08.105 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:08.150 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:08.150 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:08.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:08.188+0000 7f5746942a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:08.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:08.192+0000 7f5746942a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:08.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:08.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:01:08.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:08.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:08Z" 2024-06-15T09:01:08.216 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:08.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:01:13.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:13.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:13.219 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:13.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:13.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:13.335 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:13.332+0000 7fcbef763a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:13.336 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:13.336+0000 7fcbef763a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:13.358 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:13.359 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:13.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:13.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:13.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:13Z" 2024-06-15T09:01:13.363 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:13.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:01:18.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:18.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:18.365 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:18.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:18.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:18.456+0000 7f73d3c95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:18.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:18.456+0000 7f73d3c95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:18.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:18.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:18.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:01:18.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:18.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:18Z" 2024-06-15T09:01:18.481 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:18.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:01:23.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:23.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:23.484 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:23.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:23.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:23.595 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:23.592+0000 7ffa6b40aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:23.596 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:23.596+0000 7ffa6b40aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:23.615 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:23.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:23.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:23.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:23Z" 2024-06-15T09:01:23.620 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:23.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:01:28.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:28.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:28.622 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:28.665 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:28.665 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:28.725 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:28.724+0000 7f4f5ddcaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:28.726 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:28.724+0000 7f4f5ddcaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:28.753 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:28.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:28.753 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:28Z" 2024-06-15T09:01:28.753 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:28.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:01:33.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:33.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:33.756 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:33.798 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:33.798 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:33.851 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:33.848+0000 7f09f86aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:33.852 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:33.852+0000 7f09f86aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:33.870 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:33.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:33.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:33.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:33.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:33Z" 2024-06-15T09:01:33.875 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:33.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:01:38.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:38.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:38.878 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:38.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:38.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:38.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:38.972+0000 7f5236e33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:38.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:38.972+0000 7f5236e33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:39.015 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:39.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:39.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:01:39.018 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:39.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:38Z" 2024-06-15T09:01:39.020 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:39.021 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:01:44.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:44.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:44.023 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:44.065 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:44.065 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:44.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:44.120+0000 7f6585be4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:44.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:44.124+0000 7f6585be4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:44.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:44.148 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:44.148 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:44.148 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:44.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:44.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:01:44.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:44.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:44.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:44.154 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:44.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:44.154 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:44Z" 2024-06-15T09:01:44.154 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:44.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:01:49.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:49.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:49.156 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:49.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:49.203 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:49.261 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:49.259+0000 7f038ccbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:49.261 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:49.259+0000 7f038ccbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:49.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:49.291 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:49Z" 2024-06-15T09:01:49.291 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:49.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:01:54.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:54.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:54.293 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:54.338 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:54.338 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:54.392 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:54.387+0000 7f18d7184a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:54.393 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:54.391+0000 7f18d7184a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:54.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:54.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:54.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:54.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:54.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:54Z" 2024-06-15T09:01:54.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:54.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.699165_557.1 2024-06-15T09:01:59.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:59.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:01:59.419 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:01:59.462 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:01:59.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:01:59.514 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:59.511+0000 7f8fd787da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:59.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:01:59.511+0000 7f8fd787da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:59.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:59.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:01:59.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:01:59.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:01:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:01:59.542 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:01:59Z" 2024-06-15T09:01:59.542 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:01:59.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.699165_557.1 2024-06-15T09:02:04.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:04.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:04.544 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:04.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:04.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:04.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:04.627+0000 7f70eed8ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:04.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:04.627+0000 7f70eed8ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:04.651 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:04.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:04.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:04.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:04.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:04.656 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:04.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:04.657 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:04Z" 2024-06-15T09:02:04.657 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:04.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:02:09.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:09.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:09.660 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:09.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:09.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:09.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:09.743+0000 7fd1d841ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:09.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:09.743+0000 7fd1d841ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:09.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:09.767 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:09.768 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:09Z" 2024-06-15T09:02:09.769 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:09.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:02:14.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:14.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:14.772 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:14.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:14.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:14.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:14.859+0000 7f0027639a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:14.863 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:14.859+0000 7f0027639a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:14.903 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:14.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:14.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:14.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:14.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:14.909 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:14.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:14.909 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:14Z" 2024-06-15T09:02:14.909 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:14.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:02:19.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:19.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:19.911 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:19.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:19.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:20.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:20.011+0000 7fe689693a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:20.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:20.015+0000 7fe689693a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:20.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:20.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:20.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:20.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:20.046 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:20Z" 2024-06-15T09:02:20.046 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:20.047 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.742181_561.1 2024-06-15T09:02:25.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:25.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:25.048 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:25.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:25.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:25.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:25.151+0000 7f4ca1378a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:25.154 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:25.151+0000 7f4ca1378a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:25.189 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:25.189 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:25.189 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:25.189 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:25.189 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:25.190 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:25.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.707204_558.1", 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:25.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.707204Z", 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:25.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:25Z" 2024-06-15T09:02:25.194 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:25.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.707204_558.1 2024-06-15T09:02:30.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:30.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:30.196 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:30.239 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:30.239 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:30.293 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:30.291+0000 7f2fb91b0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:30.294 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:30.291+0000 7f2fb91b0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:30.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:30.317 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:30.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.083531_582.1", 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.083531Z", 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:30.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:30Z" 2024-06-15T09:02:30.320 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:30.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438539.083531_582.1 2024-06-15T09:02:35.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:35.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:35.322 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:35.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:35.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:35.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:35.419+0000 7f2b87a48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:35.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:35.423+0000 7f2b87a48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:35.447 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:35.447 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:35.447 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:35.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:35.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:02:35.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:35.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:35Z" 2024-06-15T09:02:35.452 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:35.453 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:02:40.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:40.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:40.455 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:40.497 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:40.498 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:40.550 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:40.546+0000 7f81b6fb9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:40.551 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:40.546+0000 7f81b6fb9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:40.570 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:40.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:40.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:40.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:40.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:40Z" 2024-06-15T09:02:40.575 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:40.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:02:45.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:45.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:45.577 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:45.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:45.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:45.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:45.662+0000 7f13fd050a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:45.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:45.662+0000 7f13fd050a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:45.685 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:45.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:45.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:02:45.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:45Z" 2024-06-15T09:02:45.690 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:45.691 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:02:50.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:50.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:50.693 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:50.737 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:50.738 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:50.782 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:50.778+0000 7f6405ad6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:50.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:50.778+0000 7f6405ad6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:50.802 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:50.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:50.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:50.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:50.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:50Z" 2024-06-15T09:02:50.807 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:50.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.699165_557.1 2024-06-15T09:02:55.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:55.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:02:55.809 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:02:55.852 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:02:55.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:02:55.900 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:55.898+0000 7f7a20079a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:55.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:02:55.898+0000 7f7a20079a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:55.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:55.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:02:55.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.699165_557.1", 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.699165Z", 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:55.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.929 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:02:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:02:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:02:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:02:55.930 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:02:55Z" 2024-06-15T09:02:55.930 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:02:55.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.699165_557.1 2024-06-15T09:03:00.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:00.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:00.932 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:00.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:00.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:01.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:01.014+0000 7fc672887a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:01.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:01.014+0000 7fc672887a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:01.037 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:01.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:01.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:01.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:01.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:01.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:01.043 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:01Z" 2024-06-15T09:03:01.043 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:01.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:03:06.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:06.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:06.044 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:06.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:06.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:06.168 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:06.166+0000 7f740ff83a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:06.169 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:06.166+0000 7f740ff83a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:06.188 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:06.188 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:06.189 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:06.190 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:06.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:06.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:06.193 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:06.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:06.194 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:06Z" 2024-06-15T09:03:06.194 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:06.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:11.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:11.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:11.196 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:11.239 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:11.240 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:11.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:11.286+0000 7f5154785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:11.292 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:11.290+0000 7f5154785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:11.311 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:11.311 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:11.311 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:11.311 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:11.311 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:11.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:11.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:11.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:11.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:11Z" 2024-06-15T09:03:11.316 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:11.317 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:16.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:16.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:16.319 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:16.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:16.363 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:16.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:16.402+0000 7f3efe8aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:16.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:16.402+0000 7f3efe8aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:16.425 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:16.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:16.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:16.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:16.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:16Z" 2024-06-15T09:03:16.430 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:16.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:21.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:21.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:21.433 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:21.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:21.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:21.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:21.538+0000 7fd84185fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:21.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:21.538+0000 7fd84185fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:21.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:21.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:21.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:21.564 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:21Z" 2024-06-15T09:03:21.564 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:21.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:26.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:26.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:26.566 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:26.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:26.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:26.652 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:26.650+0000 7f28121e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:26.652 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:26.654+0000 7f28121e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:26.671 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:26Z" 2024-06-15T09:03:26.676 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:26.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:31.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:31.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:31.678 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:31.722 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:31.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:31.780 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:31.777+0000 7fd9bdc36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:31.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:31.781+0000 7fd9bdc36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:31.800 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:31.800 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:31.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:31.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:31.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:31.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:31.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:31.806 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:31.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:31.806 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:31Z" 2024-06-15T09:03:31.806 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:31.807 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:36.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:36.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:36.808 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:36.856 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:36.856 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:36.913 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:36.913+0000 7fd32b9f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:36.914 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:36.913+0000 7fd32b9f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:36.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:36.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:36.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:36.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:36.939 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:36.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:36.939 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:36Z" 2024-06-15T09:03:36.939 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:36.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:41.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:41.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:41.941 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:41.984 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:41.985 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:42.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:42.045+0000 7fd2dadaca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:42.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:42.045+0000 7fd2dadaca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:42.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:42.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:03:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:42Z" 2024-06-15T09:03:42.077 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:42.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:03:47.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:47.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:47.080 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:47.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:47.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:47.169 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:47.169+0000 7fd9b97c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:47.170 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:47.169+0000 7fd9b97c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:47.194 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:47.194 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:47.194 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:47.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:47.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:47.197 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:47.198 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:47Z" 2024-06-15T09:03:47.199 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:47.200 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:03:52.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:52.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:52.201 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:52.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:52.245 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:52.287 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:52.285+0000 7fe6bec33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:52.288 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:52.289+0000 7fe6bec33a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:52.307 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:52.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:52.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:03:52.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:52.311 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:52Z" 2024-06-15T09:03:52.312 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:52.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:03:57.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:57.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:03:57.314 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:03:57.357 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:03:57.357 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:03:57.416 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:57.413+0000 7f8a2ca97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:57.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:03:57.417+0000 7f8a2ca97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.440 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:03:57.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:57.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:03:57.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:03:57Z" 2024-06-15T09:03:57.444 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:03:57.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:02.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:02.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:02.447 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:02.490 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:02.490 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:02.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:02.541+0000 7f5dc266fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:02.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:02.541+0000 7f5dc266fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:02.577 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:02.577 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:02.577 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:02.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:02.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:02.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:02.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:02Z" 2024-06-15T09:04:02.582 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:02.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:07.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:07.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:07.586 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:07.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:07.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:07.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:07.681+0000 7ff2b5401a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:07.683 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:07.681+0000 7ff2b5401a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:07.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:07.704 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:07.705 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:07.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:07.706 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:07Z" 2024-06-15T09:04:07.706 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:07.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:12.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:12.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:12.708 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:12.753 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:12.753 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:12.794 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:12.793+0000 7fce3ccf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:12.795 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:12.793+0000 7fce3ccf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:12.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:12.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:12.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:12.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:12.823 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:12.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:12.823 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:12Z" 2024-06-15T09:04:12.823 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:12.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:17.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:17.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:17.825 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:17.870 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:17.870 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:17.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:17.929+0000 7f9d4e44da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:17.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:17.929+0000 7f9d4e44da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:17.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:17.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:17.959 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:17Z" 2024-06-15T09:04:17.960 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:17.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:22.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:22.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:22.962 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:23.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:23.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:23.056 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:23.056+0000 7f256b6e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:23.057 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:23.056+0000 7f256b6e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:23Z" 2024-06-15T09:04:23.080 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:23.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438539.261071_591.1 2024-06-15T09:04:28.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:28.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:28.083 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:28.127 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:28.127 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:28.170 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:28.168+0000 7f62647efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:28.171 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:28.168+0000 7f62647efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:28.189 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:28.190 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:28.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:28.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:28Z" 2024-06-15T09:04:28.193 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:28.194 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438537.989943_579.1 2024-06-15T09:04:33.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:33.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:33.196 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:33.243 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:33.244 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:33.297 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:33.296+0000 7f262d06ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:33.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:33.296+0000 7f262d06ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:33.321 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:33.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:33.321 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:33.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:33Z" 2024-06-15T09:04:33.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:33.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:38.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:38.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:38.328 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:38.374 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:38.374 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:38.445 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:38.444+0000 7f302e58fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:38.449 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:38.448+0000 7f302e58fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:38.476 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:38.476 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:38.476 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:38Z" 2024-06-15T09:04:38.481 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:38.482 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:43.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:43.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:43.483 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:43.527 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:43.527 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:43.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:43.576+0000 7f8ba14a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:43.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:43.576+0000 7f8ba14a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:43.603 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:43.603 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:43.603 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:43.603 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:43.604 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:43.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:43.606 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:43.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:43.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:43.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:43.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:43.611 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:43Z" 2024-06-15T09:04:43.611 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:43.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438537.989943_579.1 2024-06-15T09:04:48.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:48.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:48.614 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:48.663 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:48.663 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:48.737 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:48.736+0000 7fbc9f95fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:48.738 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:48.736+0000 7fbc9f95fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:48.758 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:48Z" 2024-06-15T09:04:48.763 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:48.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.728887_559.1 2024-06-15T09:04:53.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:53.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:53.765 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:53.809 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:53.809 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:53.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:53.860+0000 7f2f378f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:53.861 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:53.860+0000 7f2f378f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:53.881 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:53.881 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:53.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:53.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:53.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:53.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:53.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:53.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:53.888 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:53.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:53.888 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:53Z" 2024-06-15T09:04:53.888 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:53.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718441990.938224_1201.1 target=1_1718438536.690655_555.1 2024-06-15T09:04:58.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-06-15T09:04:58.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:58.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:58.939 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:59.034 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:59.034 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:59.178 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T09:04:59.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718442298.934048_1222.1", 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:04:58.934048Z" 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:04:59.242 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:04:59.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:04:59.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:59.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:04:59.245 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:04:59.290 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:04:59.290 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:04:59.348 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:59.348+0000 7f3f63b7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:59.349 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:04:59.348+0000 7f3f63b7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:04:59.368 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:04:59.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:04:59.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:59.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:59.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:04:59Z" 2024-06-15T09:04:59.373 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:04:59.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.728887_559.1 2024-06-15T09:05:04.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:04.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:04.376 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:04.419 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:04.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:04.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:04.480+0000 7ffba7e19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:04.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:04.480+0000 7ffba7e19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:04.506 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:05:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:04.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:04.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.513 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:04.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:04.514 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:04.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:04.514 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:04Z" 2024-06-15T09:05:04.514 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:04.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.728887_559.1 2024-06-15T09:05:09.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:09.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:09.517 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:09.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:09.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:09.621 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:09.620+0000 7f897cd12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:09.622 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:09.620+0000 7f897cd12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:09.647 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:09.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:09.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:09.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:05:09.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:09.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:09.653 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:09Z" 2024-06-15T09:05:09.654 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:09.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.728887_559.1 2024-06-15T09:05:14.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:14.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:14.658 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:14.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:14.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:14.749 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:14.748+0000 7f8f39165a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:14.750 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:14.748+0000 7f8f39165a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:14.769 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:14.769 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:14.769 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:14.769 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:14.769 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:14.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:14.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:14.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:14.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:14Z" 2024-06-15T09:05:14.774 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:14.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.690655_555.1 2024-06-15T09:05:19.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:19.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:19.777 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:19.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:19.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:19.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:19.863+0000 7fc8cca05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:19.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:19.863+0000 7fc8cca05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:19.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.690655_555.1", 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.690655Z", 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:19.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:19.888 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:19.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:19.889 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:19Z" 2024-06-15T09:05:19.889 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:19.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.690655_555.1 2024-06-15T09:05:24.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:24.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:24.891 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:24.935 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:24.935 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:25.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:25.011+0000 7f9900088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:25.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:25.011+0000 7f9900088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:25.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:25.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.728887_559.1", 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:25.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.728887Z", 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:25.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:25Z" 2024-06-15T09:05:25.071 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:25.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.728887_559.1 2024-06-15T09:05:30.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:30.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:30.074 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:30.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:30.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:30.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:30.159+0000 7f3e2eee0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:30.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:30.159+0000 7f3e2eee0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:30.181 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:30.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:30.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:30.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:30.186 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:30Z" 2024-06-15T09:05:30.187 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:30.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.742181_561.1 2024-06-15T09:05:35.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:35.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:35.190 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:35.233 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:35.233 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:35.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:35.279+0000 7f3750b56a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:35.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:35.279+0000 7f3750b56a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:35.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:35.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:35.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:35.312 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:35Z" 2024-06-15T09:05:35.313 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:35.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.742181_561.1 2024-06-15T09:05:40.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:40.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:40.314 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:40.360 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:40.360 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:40.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:40.399+0000 7f04528efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:40.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:40.399+0000 7f04528efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:40.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:40.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:05:40.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:40.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:40Z" 2024-06-15T09:05:40.427 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:40.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.742181_561.1 2024-06-15T09:05:45.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:45.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:45.430 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:45.475 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:45.475 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:45.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:45.535+0000 7fc98fe95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:45.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:45.535+0000 7fc98fe95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:45.560 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:45.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:45.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:05:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:45.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:45.565 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:45Z" 2024-06-15T09:05:45.566 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:45.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.742181_561.1 2024-06-15T09:05:50.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:50.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:50.568 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:50.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:50.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:50.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:50.659+0000 7fa6c4c19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:50.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:50.663+0000 7fa6c4c19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:50.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:50.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.765305_564.1", 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.765305Z", 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:50.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:50Z" 2024-06-15T09:05:50.692 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:50.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.765305_564.1 2024-06-15T09:05:55.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:55.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:05:55.695 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:05:55.738 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:05:55.738 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:05:55.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:55.787+0000 7fc5732b9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:55.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:05:55.787+0000 7fc5732b9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:55.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:55.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T09:05:55.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:05:55.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:05:55Z" 2024-06-15T09:05:55.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:05:55.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.759965_563.1 2024-06-15T09:06:00.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:00.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:00.816 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:00.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:00.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:00.913 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:00.911+0000 7f9381099a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:00.914 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:00.911+0000 7f9381099a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:00.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:00.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:00.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:00.938 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:00.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:00.939 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:00Z" 2024-06-15T09:06:00.939 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:00.940 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.955581_573.1 2024-06-15T09:06:05.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:05.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:05.941 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:05.984 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:05.984 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:06.026 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:06.023+0000 7f97f6239a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:06.027 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:06.023+0000 7f97f6239a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:06.045 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:06.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:06.050 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:06.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:06.050 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:06Z" 2024-06-15T09:06:06.050 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:06.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:11.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:11.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:11.052 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:11.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:11.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:11.144 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:11.142+0000 7ff47c76fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:11.145 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:11.142+0000 7ff47c76fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:11.163 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:11.163 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:11Z" 2024-06-15T09:06:11.168 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:11.169 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:16.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:16.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:16.170 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:16.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:16.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:16.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:16.250+0000 7fd42d416a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:16.254 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:16.250+0000 7fd42d416a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:16.272 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:16.272 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:16.272 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:16.272 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:16.272 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:16.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:16.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:16Z" 2024-06-15T09:06:16.277 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:16.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:21.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:21.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:21.279 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:21.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:21.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:21.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:21.374+0000 7ff80912aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:21.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:21.378+0000 7ff80912aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:21.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:21.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:21.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:21.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:21.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:21Z" 2024-06-15T09:06:21.410 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:21.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:26.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:26.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:26.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:26.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:26.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:26.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:26.510+0000 7fb309632a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:26.516 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:26.514+0000 7fb309632a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:26.538 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:26.538 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:26.538 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:26.538 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:26.538 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:26.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:26.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.542 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:26Z" 2024-06-15T09:06:26.543 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:26.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:31.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:31.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:31.546 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:31.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:31.589 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:31.634 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:31.630+0000 7fce6a0daa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:31.635 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:31.630+0000 7fce6a0daa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:31.653 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:31.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:31.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:31.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:31.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:31Z" 2024-06-15T09:06:31.658 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:31.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:36.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:36.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:36.661 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:36.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:36.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:36.753 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:36.750+0000 7f9fd0977a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:36.754 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:36.750+0000 7f9fd0977a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:36.773 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:36.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:36.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:36.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:36.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:36.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:36.779 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:36.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:36.779 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:36Z" 2024-06-15T09:06:36.779 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:36.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:41.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:41.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:41.781 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:41.834 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:41.835 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:41.900 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:41.898+0000 7f5de9e08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:41.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:41.898+0000 7f5de9e08a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:41.925 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:41.925 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:41.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:41.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:41Z" 2024-06-15T09:06:41.930 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:41.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:46.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:46.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:46.932 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:46.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:46.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:47.024 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:47.022+0000 7f70e4662a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:47.025 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:47.022+0000 7f70e4662a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:47.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:47.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:47.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:47.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:47.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:47Z" 2024-06-15T09:06:47.051 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:47.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:52.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:52.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:52.052 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:52.095 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:52.095 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:52.139 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:52.138+0000 7fa81f7d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:52.140 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:52.142+0000 7fa81f7d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:52.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:52.159 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:52.159 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:52.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:52.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:52.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:52.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:52Z" 2024-06-15T09:06:52.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:52.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:06:57.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:57.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:06:57.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:06:57.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:06:57.211 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:06:57.263 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:57.262+0000 7f9f65efba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:57.264 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:06:57.266+0000 7f9f65efba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:06:57.294 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:06:57.294 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:57.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:06:57.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:06:57.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:06:57Z" 2024-06-15T09:06:57.299 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:06:57.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:02.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:02.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:02.301 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:02.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:02.345 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:02.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:02.401+0000 7f9650c00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:02.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:02.405+0000 7f9650c00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:02.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:02.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:02.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:02.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:02Z" 2024-06-15T09:07:02.427 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:02.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:07.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:07.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:07.430 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:07.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:07.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:07.513 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:07.513+0000 7fb7943f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:07.514 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:07.513+0000 7fb7943f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:07.532 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:07.532 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:07.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:07.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:07.536 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:07Z" 2024-06-15T09:07:07.537 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:07.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:07:12.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:12.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:12.540 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:12.583 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:12.583 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:12.645 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:12.645+0000 7f8c87227a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:12.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:12.645+0000 7f8c87227a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:12.667 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:12.667 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:12.667 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:12.667 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:12.667 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:12.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:12.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:07:12.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:12.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:12Z" 2024-06-15T09:07:12.672 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:12.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:07:17.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:17.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:17.674 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:17.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:17.717 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:17.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:17.765+0000 7f4eb7d4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:17.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:17.765+0000 7f4eb7d4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:17.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:17.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:17.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:17.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:17Z" 2024-06-15T09:07:17.795 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:17.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:22.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:22.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:22.798 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:22.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:22.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:22.884 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:22.885+0000 7f56f2070a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:22.885 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:22.885+0000 7f56f2070a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:22.905 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:22.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:22.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:22.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:22.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:22.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:22.911 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:22Z" 2024-06-15T09:07:22.911 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:22.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:27.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:27.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:27.912 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:27.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:27.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:28.006 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:28.005+0000 7ffb92426a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:28.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:28.005+0000 7ffb92426a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:28.028 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:28.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:28.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.030 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:28.030 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:28.030 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:28.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:28.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:28.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:28.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:28.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:28.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:28.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:28.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:28.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:28.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:28.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:28.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:28.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:28.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:28.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:28.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:28.037 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:28.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:28.038 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:28Z" 2024-06-15T09:07:28.038 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:28.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:33.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:33.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:33.040 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:33.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:33.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:33.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:33.125+0000 7fa63f827a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:33.126 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:33.125+0000 7fa63f827a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:33.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:33.148 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:33Z" 2024-06-15T09:07:33.149 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:33.150 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:38.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:38.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:38.151 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:38.194 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:38.194 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:38.236 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:38.237+0000 7f4a5213ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:38.238 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:38.237+0000 7f4a5213ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:38.302 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:38.302 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:38.302 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:38.302 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:38.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:38.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:38.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:38.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:38Z" 2024-06-15T09:07:38.307 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:38.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:43.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:43.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:43.309 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:43.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:43.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:43.411 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:43.409+0000 7fbf6e4bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:43.412 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:43.413+0000 7fbf6e4bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:43.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:43.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:43.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:43.448 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:43.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:43.448 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:43Z" 2024-06-15T09:07:43.448 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:43.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:07:48.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:48.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:48.454 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:48.498 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:48.498 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:48.553 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:48.553+0000 7f4f021fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:48.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:48.553+0000 7f4f021fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:48.574 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:48.574 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:48.574 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:48.574 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:48.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:48.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:48.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:48.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:48Z" 2024-06-15T09:07:48.579 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:48.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.832707_565.1 2024-06-15T09:07:53.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:53.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:53.582 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:53.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:53.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:53.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:53.664+0000 7f6dfe216a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:53.668 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:53.668+0000 7f6dfe216a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:53.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:53.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:53.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:53.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:53Z" 2024-06-15T09:07:53.693 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:53.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.920756_572.1 2024-06-15T09:07:58.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:58.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:07:58.696 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:07:58.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:07:58.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:07:58.793 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:58.792+0000 7f6010e3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:58.794 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:07:58.792+0000 7f6010e3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.814 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:58.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:07:58.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:07:58.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:07:58Z" 2024-06-15T09:07:58.819 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:07:58.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:08:03.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:03.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:03.821 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:03.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:03.866 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:03.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:03.904+0000 7f25f7a8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:03.908 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:03.908+0000 7f25f7a8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:03.929 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:03.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:03.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:03.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:03.934 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:03.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:03.934 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:03Z" 2024-06-15T09:08:03.934 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:03.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:08:08.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:08.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:08.936 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:08.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:08.979 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:09.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:09.036+0000 7f3bad624a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:09.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:09.040+0000 7f3bad624a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:09.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:09.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:08:09.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:09.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:09Z" 2024-06-15T09:08:09.065 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:09.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:08:14.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:14.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:14.068 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:14.110 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:14.111 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:14.169 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:14.168+0000 7f1cfa69ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:14.170 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:14.168+0000 7f1cfa69ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:14.189 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:14.190 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:14.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:08:14.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:14.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:14Z" 2024-06-15T09:08:14.194 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:14.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:08:19.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:19.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:19.197 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:19.239 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:19.240 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:19.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:19.296+0000 7ff47fa32a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:19.303 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:19.300+0000 7ff47fa32a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:19.375 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:19.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:19.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:19.378 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:19.379 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:19Z" 2024-06-15T09:08:19.380 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:19.381 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.955581_573.1 2024-06-15T09:08:24.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:24.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:24.383 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:24.425 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:24.426 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:24.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:24.476+0000 7fdcaede4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:24.480 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:24.480+0000 7fdcaede4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:24.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:24.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:24.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:24.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:24.507 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:24Z" 2024-06-15T09:08:24.507 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:24.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:08:29.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:29.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:29.509 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:29.552 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:29.553 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:29.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:29.612+0000 7f31bc60ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:29.730 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:29.612+0000 7f31bc60ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:29.730 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:29.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:29.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:08:29.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:29.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:29Z" 2024-06-15T09:08:29.735 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:29.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:08:34.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:34.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:34.737 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:34.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:34.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:34.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:34.820+0000 7fb7253f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:34.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:34.820+0000 7fb7253f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:34.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:34.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:34Z" 2024-06-15T09:08:34.848 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:34.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:08:39.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:39.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:39.851 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:39.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:39.895 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:39.944 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:39.944+0000 7f7f798d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:39.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:39.952+0000 7f7f798d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.990 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:39.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:39Z" 2024-06-15T09:08:39.995 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:39.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.742181_561.1 2024-06-15T09:08:44.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:44.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:44.998 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:45.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:45.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:45.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:45.095+0000 7f4ca9af2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:45.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:45.095+0000 7f4ca9af2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:45.118 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:45.118 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:45.118 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:45.118 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:45.118 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:45.119 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:45.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:45.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:45Z" 2024-06-15T09:08:45.123 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:45.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:08:50.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:50.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:50.125 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:50.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:50.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:50.218 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:50.215+0000 7fa52a5dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:50.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:50.215+0000 7fa52a5dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:50.247 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:50Z" 2024-06-15T09:08:50.247 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:50.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.832707_565.1 2024-06-15T09:08:55.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:55.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:08:55.262 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:08:55.311 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:08:55.311 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:08:55.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:55.363+0000 7ff23e47ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:55.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:08:55.363+0000 7ff23e47ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:08:55.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:55.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:55.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:08:55.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:08:55.446 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:08:55.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:08:55.447 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:08:55Z" 2024-06-15T09:08:55.447 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:08:55.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.832707_565.1 2024-06-15T09:09:00.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:00.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:00.449 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:00.493 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:00.493 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:00.545 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:00.543+0000 7f4519fd4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:00.546 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:00.543+0000 7f4519fd4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:00.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:00.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:00.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:00.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:00.570 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:00Z" 2024-06-15T09:09:00.571 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:00.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.979596_577.1 2024-06-15T09:09:05.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:05.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:05.573 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:05.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:05.624 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:05.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:05.679+0000 7ffa8e452a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:05.683 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:05.679+0000 7ffa8e452a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.700 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:05.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:05.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:05.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:05.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:05.705 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:05.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:05.705 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:05Z" 2024-06-15T09:09:05.705 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:05.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:10.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:10.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:10.707 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:10.750 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:10.750 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:10.794 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:10.791+0000 7f165f997a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:10.795 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:10.791+0000 7f165f997a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:10.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:10.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:10.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:10.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:10Z" 2024-06-15T09:09:10.820 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:10.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:15.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:15.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:15.823 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:15.863 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:15.864 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:15.915 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:15.915+0000 7ff37ffb7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:15.916 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:15.915+0000 7ff37ffb7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:15.937 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:15.937 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:15.937 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:15.937 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:15.937 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:15.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:15.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:15.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:15.941 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:15Z" 2024-06-15T09:09:15.942 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:15.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:20.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:20.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:20.944 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:20.987 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:20.987 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:21.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:21.043+0000 7f5f2fd6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:21.047 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:21.043+0000 7f5f2fd6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.066 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:21.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:21.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:21.070 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:21.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:21.071 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:21Z" 2024-06-15T09:09:21.071 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:21.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:26.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:26.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:26.073 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:26.116 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:26.116 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:26.172 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:26.171+0000 7f36a9136a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:26.173 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:26.171+0000 7f36a9136a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:26.191 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:26.191 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:26.191 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:26.191 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:26.192 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:26.192 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:26.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:26.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.192 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:26.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:26.194 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:26.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:26.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:26.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:26.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:26.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:26.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:26.197 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:26.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:26.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:26.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:26.201 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:26.202 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:26Z" 2024-06-15T09:09:26.202 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:26.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:31.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:31.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:31.205 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:31.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:31.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:31.308 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:31.307+0000 7f8374d5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:31.309 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:31.307+0000 7f8374d5ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:31.330 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:31.330 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:31.330 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:31.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:31.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:31.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:31.334 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:31Z" 2024-06-15T09:09:31.335 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:31.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.920756_572.1 2024-06-15T09:09:36.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:36.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:36.337 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:36.379 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:36.379 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:36.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:36.418+0000 7f7503b98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:36.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:36.422+0000 7f7503b98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:36.444 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:36.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:36.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:36.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:36.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:36.449 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:36Z" 2024-06-15T09:09:36.450 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:36.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.920756_572.1 2024-06-15T09:09:41.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:41.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:41.452 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:41.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:41.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:41.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:41.566+0000 7f51f3695a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:41.571 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:41.566+0000 7f51f3695a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:41.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:41.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:41.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:41.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:41Z" 2024-06-15T09:09:41.600 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:41.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.832707_565.1 2024-06-15T09:09:46.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:46.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:46.603 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:46.647 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:46.647 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:46.692 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:46.690+0000 7f113b2d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:46.693 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:46.690+0000 7f113b2d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:46.711 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:46.711 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:46.711 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:46.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:46.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:46.714 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:46.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:46Z" 2024-06-15T09:09:46.716 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:46.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:51.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:51.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:51.719 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:51.776 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:51.777 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:51.837 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:51.834+0000 7f516e682a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:51.837 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:51.834+0000 7f516e682a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:51.869 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:51.870 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:51.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:51.872 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:51.873 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.874 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:51.875 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.876 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:51.877 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:51.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:51.879 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:51.880 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:51.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:51.880 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:51.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:51.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:51.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:51.882 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:51.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:51.882 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:51Z" 2024-06-15T09:09:51.882 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:51.884 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:09:56.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:56.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:09:56.886 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:09:56.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:09:56.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:09:56.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:56.982+0000 7f8ece9e3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:56.985 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:09:56.982+0000 7f8ece9e3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:09:57.007 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:09:57.007 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:09:57.007 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:09:57.007 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:09:57.007 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:57.008 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:57.009 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:57.010 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:09:57.011 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:09:56Z" 2024-06-15T09:09:57.012 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:09:57.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:02.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:02.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:02.014 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:02.057 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:02.057 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:02.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:02.118+0000 7f72f78cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:02.120 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:02.118+0000 7f72f78cca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:02.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:02.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:02.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:02.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:02.149 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:02Z" 2024-06-15T09:10:02.150 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:02.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442298.934048_1222.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:07.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-06-15T09:10:07.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:07.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:07.204 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:07.247 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:07.409 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:10:07.485 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718442607.199216_1254.1", 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:10:07.199216Z" 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:10:07.486 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:10:07.487 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:10:07.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:07.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:07.488 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:07.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:07.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:07.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:07.590+0000 7f1e26182a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:07.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:07.590+0000 7f1e26182a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:07.619 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:07.619 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:07.619 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:07.619 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:07.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:07.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:07.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:07.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:07Z" 2024-06-15T09:10:07.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:07.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:12.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:12.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:12.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:12.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:12.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:12.726 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:12.722+0000 7f01aa100a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:12.727 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:12.726+0000 7f01aa100a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:12.747 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:12.747 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:12.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:12.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:12.750 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.751 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:12.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:12Z" 2024-06-15T09:10:12.753 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:12.754 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:10:17.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:17.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:17.756 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:17.798 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:17.799 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:17.843 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:17.842+0000 7fb571212a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:17.844 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:17.846+0000 7fb571212a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:17.862 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:17.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:17.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:17Z" 2024-06-15T09:10:17.867 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:17.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:22.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:22.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:22.870 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:22.913 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:22.913 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:22.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:22.974+0000 7f7c04f71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:22.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:22.974+0000 7f7c04f71a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:23.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:23.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:23.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:23.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:23.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:23.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:22Z" 2024-06-15T09:10:23.016 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:23.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:28.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:28.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:28.018 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:28.061 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:28.061 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:28.122 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:28.121+0000 7fa4bc69ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:28.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:28.125+0000 7fa4bc69ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:28.146 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:28.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:28.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:28.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:28.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:28Z" 2024-06-15T09:10:28.151 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:28.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:10:33.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:33.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:33.153 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:33.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:33.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:33.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:33.321+0000 7f402ba0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:33.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:33.321+0000 7f402ba0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:33.339 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:33.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:33.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:33.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:33.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:33Z" 2024-06-15T09:10:33.344 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:33.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:38.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:38.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:38.347 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:38.393 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:38.393 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:38.450 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:38.449+0000 7f571b1e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:38.451 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:38.453+0000 7f571b1e4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:38.477 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:38.477 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:38.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:38.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:38.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:38.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:38.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:38.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:38.484 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:38.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:38.485 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:38Z" 2024-06-15T09:10:38.485 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:38.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:43.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:43.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:43.488 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:43.576 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:43.577+0000 7f940052da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:43.577 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:43.577+0000 7f940052da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:43.595 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:43.595 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:43.595 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:43.595 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:43.595 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:43.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:43.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:43.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:43.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:43.600 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:43Z" 2024-06-15T09:10:43.600 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:43.600 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:48.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:48.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:48.601 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:48.646 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:48.646 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:48.702 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:48.701+0000 7f7d59e09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:48.703 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:48.701+0000 7f7d59e09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:48.729 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:48.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:48.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:48.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:48Z" 2024-06-15T09:10:48.733 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:48.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:10:53.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:53.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:53.735 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:53.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:53.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:53.826 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:53.825+0000 7fad02999a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:53.827 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:53.829+0000 7fad02999a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.847 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:53.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:53.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:53.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:53Z" 2024-06-15T09:10:53.851 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:53.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:10:58.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:58.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:10:58.853 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:10:58.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:10:58.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:10:58.980 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:58.981+0000 7fb03dd9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:58.981 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:10:58.981+0000 7fb03dd9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:10:59.001 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:10:59.001 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:10:59.001 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:59.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:10:59.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:59.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:10:59.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:10:58Z" 2024-06-15T09:10:59.006 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:10:59.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.920756_572.1 2024-06-15T09:11:04.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:04.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:04.008 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:04.051 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:04.051 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:04.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:04.097+0000 7f6a441dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:04.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:04.097+0000 7f6a441dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:04.124 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:04.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:04.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:04.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:04.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:04Z" 2024-06-15T09:11:04.129 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:04.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:11:09.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:09.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:09.131 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:09.171 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:09.172 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:09.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:09.229+0000 7f8ecfa87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:09.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:09.229+0000 7f8ecfa87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:09.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:09.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:09.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:09Z" 2024-06-15T09:11:09.253 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:09.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:11:14.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:14.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:14.256 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:14.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:14.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:14.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:14.360+0000 7fb6753efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:14.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:14.360+0000 7fb6753efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:14.380 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:14.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:14.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:14.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:14.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:14.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:14.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:14.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:14.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:14.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:11:14.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:14.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:14.389 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:14.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:14.390 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:14Z" 2024-06-15T09:11:14.390 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:14.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:11:19.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:19.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:19.393 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:19.436 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:19.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:19.499 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:19.496+0000 7f4262d35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:19.499 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:19.500+0000 7f4262d35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:19.518 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:19.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:19.518 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:19.518 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:19.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:19.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:19.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:19Z" 2024-06-15T09:11:19.523 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:19.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:11:24.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:24.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:24.525 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:24.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:24.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:24.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:24.612+0000 7f4869be0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:24.614 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:24.612+0000 7f4869be0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:24.636 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:24.636 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:24.636 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:24.636 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:24.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:24.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:11:24.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:24.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:24Z" 2024-06-15T09:11:24.641 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:24.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:11:29.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:29.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:29.644 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:29.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:29.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:29.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:29.744+0000 7f8a1190aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:29.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:29.744+0000 7f8a1190aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:29.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:29.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.903214_570.1", 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:29.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.903214Z", 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:29.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:29Z" 2024-06-15T09:11:29.775 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:29.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.903214_570.1 2024-06-15T09:11:34.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:34.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:34.778 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:34.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:34.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:34.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:34.864+0000 7fabae388a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:34.868 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:34.868+0000 7fabae388a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:34.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:34.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:34.889 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:34.890 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:34.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:34.891 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:34Z" 2024-06-15T09:11:34.891 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:34.891 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:11:39.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:39.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:39.892 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:39.936 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:39.936 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:39.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:39.976+0000 7f69d7b0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:39.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:39.976+0000 7f69d7b0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:39.997 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:39.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:39.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:40.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:40.001 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:40.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:40.002 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:39Z" 2024-06-15T09:11:40.002 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:40.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.920756_572.1 2024-06-15T09:11:45.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:45.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:45.004 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:45.047 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:45.047 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:45.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:45.104+0000 7f93a5a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:45.104 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:45.104+0000 7f93a5a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:45.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:45.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:45.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:45Z" 2024-06-15T09:11:45.129 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:45.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.920756_572.1 2024-06-15T09:11:50.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:50.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:50.131 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:50.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:50.175 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:50.221 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:50.220+0000 7f2a35789a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:50.223 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:50.220+0000 7f2a35789a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:50.241 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:50.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:50.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:50.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:50.245 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:50.246 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:50Z" 2024-06-15T09:11:50.246 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:50.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:11:55.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:55.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:11:55.248 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:11:55.295 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:11:55.296 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:11:55.367 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:55.364+0000 7f9c53355a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:55.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:11:55.368+0000 7f9c53355a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:11:55.394 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:11:55.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:11:55.394 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:11:55.394 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:11:55.394 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:11:55.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:55.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:55.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:11:55.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:11:55.399 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:11:55.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:11:55.400 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:11:55Z" 2024-06-15T09:11:55.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:11:55.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.920756_572.1 2024-06-15T09:12:00.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:00.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:00.402 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:00.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:00.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:00.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:00.500+0000 7fd71b585a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:00.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:00.500+0000 7fd71b585a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:00.521 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:00.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:00.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:00.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:00.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:00Z" 2024-06-15T09:12:00.526 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:00.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:05.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:05.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:05.528 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:05.573 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:05.573 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:05.640 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:05.639+0000 7f430aec4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:05.641 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:05.639+0000 7f430aec4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:05.671 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:05.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:05.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.675 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:05.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:05.676 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:05.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:05.676 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:05Z" 2024-06-15T09:12:05.676 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:05.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438537.989943_579.1 2024-06-15T09:12:10.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:10.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:10.679 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:10.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:10.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:10.776 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:10.775+0000 7ffa216c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:10.777 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:10.775+0000 7ffa216c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:10.796 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:10.796 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:10.796 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:10.797 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:10.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:10.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:10.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:10Z" 2024-06-15T09:12:10.801 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:10.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:15.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:15.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:15.804 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:15.848 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:15.848 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:15.903 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:15.899+0000 7fcfff589a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:15.904 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:15.903+0000 7fcfff589a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:15.923 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:15.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:15.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.964810_576.1", 2024-06-15T09:12:15.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.964810Z", 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:15.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:15Z" 2024-06-15T09:12:15.928 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:15.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.964810_576.1 2024-06-15T09:12:20.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:20.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:20.931 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:20.973 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:20.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:21.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:21.011+0000 7f5f64145a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:21.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:21.015+0000 7f5f64145a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:21.034 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:21.034 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:21.034 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:21.034 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:21.034 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:21.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:21.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:21.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:21.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:21.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:21.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:21.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:21.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:21.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:21.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:21.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:21.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:21.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:21.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:21.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:21.043 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:21.044 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:21Z" 2024-06-15T09:12:21.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:21.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:26.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:26.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:26.048 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:26.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:26.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:26.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:26.135+0000 7f16cb03da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:26.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:26.135+0000 7f16cb03da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:26.159 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:26.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:26.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:26.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:26.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:26Z" 2024-06-15T09:12:26.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:26.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:31.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:31.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:31.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:31.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:31.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:31.264 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:31.263+0000 7f0dfb93ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:31.265 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:31.263+0000 7f0dfb93ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:31.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:31.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.903214_570.1", 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.903214Z", 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:31.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:31.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:31.294 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:31Z" 2024-06-15T09:12:31.294 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:31.294 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.903214_570.1 2024-06-15T09:12:36.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:36.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:36.296 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:36.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:36.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:36.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:36.399+0000 7feac538ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:36.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:36.403+0000 7feac538ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:36.428 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:36.428 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:36.428 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:36.428 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:36.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:36.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:36.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:36.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:36Z" 2024-06-15T09:12:36.433 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:36.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:41.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:41.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:41.435 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:41.480 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:41.480 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:41.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:41.519+0000 7f37ff7e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:41.524 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:41.523+0000 7f37ff7e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:41.542 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:41.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:41.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:41Z" 2024-06-15T09:12:41.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:41.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:46.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:46.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:46.549 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:46.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:46.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:46.635 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:46.635+0000 7ff44a4bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:46.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:46.635+0000 7ff44a4bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.655 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:46.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:46.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:46.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:46Z" 2024-06-15T09:12:46.659 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:46.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:12:51.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:51.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:51.662 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:51.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:51.705 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:51.758 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:51.755+0000 7ff414814a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:51.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:51.759+0000 7ff414814a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:51.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:51.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:51.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:51Z" 2024-06-15T09:12:51.788 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:51.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:12:56.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:56.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:12:56.790 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:12:56.835 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:12:56.835 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:12:56.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:56.891+0000 7fc4c2b49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:56.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:12:56.891+0000 7fc4c2b49a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:12:56.919 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:12:56.919 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:12:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:12:56.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:56.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:12:56.923 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:12:56Z" 2024-06-15T09:12:56.924 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:12:56.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:13:01.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:01.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:01.926 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:01.969 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:01.969 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:02.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:02.018+0000 7fd60b9b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:02.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:02.018+0000 7fd60b9b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:02.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:02.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:02Z" 2024-06-15T09:13:02.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:02.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.759965_563.1 2024-06-15T09:13:07.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:07.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:07.047 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:07.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:07.090 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:07.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:07.142+0000 7f376dd15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:07.144 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:07.142+0000 7f376dd15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:07.169 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:07.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:07.171 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.964810_576.1", 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:07.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.964810Z", 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:07.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:07Z" 2024-06-15T09:13:07.174 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:07.175 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.964810_576.1 2024-06-15T09:13:12.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:12.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:12.176 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:12.217 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:12.217 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:12.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:12.274+0000 7f6e6cb9da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:12.279 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:12.274+0000 7f6e6cb9da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:12.305 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:12.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:12.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:13:12.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:12.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:12.310 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:12Z" 2024-06-15T09:13:12.311 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:12.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:13:17.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:17.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:17.313 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:17.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:17.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:17.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:17.414+0000 7fc1b213ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:17.418 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:17.414+0000 7fc1b213ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:13:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:17Z" 2024-06-15T09:13:17.441 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:17.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:13:22.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:22.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:22.443 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:22.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:22.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:22.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:22.542+0000 7f99b32fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:22.544 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:22.542+0000 7f99b32fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:22.566 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:22.566 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:22.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:22.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:22.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:22.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:22Z" 2024-06-15T09:13:22.571 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:22.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:13:27.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:27.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:27.573 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:27.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:27.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:27.660 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:27.658+0000 7f33fe150a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:27.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:27.658+0000 7f33fe150a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:27.681 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:27.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.907237_571.1", 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.907237Z", 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:27.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.685 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:27.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:27.686 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:27.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:27.686 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:27Z" 2024-06-15T09:13:27.686 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:27.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.907237_571.1 2024-06-15T09:13:32.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:32.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:32.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:32.732 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:32.732 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:32.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:32.810+0000 7fbd6ce90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:32.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:32.810+0000 7fbd6ce90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:32.837 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:32.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:32.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:32.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:32.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:32Z" 2024-06-15T09:13:32.842 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:32.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:13:37.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:37.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:37.845 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:37.887 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:37.888 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:37.931 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:37.926+0000 7f46d1b6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:37.932 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:37.930+0000 7f46d1b6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:37.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:37.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:37.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:37.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:37.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:37Z" 2024-06-15T09:13:37.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:37.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.979596_577.1 2024-06-15T09:13:42.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:42.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:42.958 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:43.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:43.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:43.057 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:43.058+0000 7f51413c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:43.058 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:43.058+0000 7f51413c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.080 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:43.081 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:43.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:43.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:43.084 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:43Z" 2024-06-15T09:13:43.085 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:43.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.979596_577.1 2024-06-15T09:13:48.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:48.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:48.087 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:48.130 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:48.130 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:48.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:48.182+0000 7f6f6f98da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:48.183 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:48.186+0000 7f6f6f98da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:48.222 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:48.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:48.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.903214_570.1", 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:48.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.903214Z", 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:48.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:48Z" 2024-06-15T09:13:48.227 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:48.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.903214_570.1 2024-06-15T09:13:53.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:53.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:53.230 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:53.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:53.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:53.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:53.333+0000 7f98a142ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:53.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:53.333+0000 7f98a142ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:53.356 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:53.356 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:53.358 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.903214_570.1", 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:53.359 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.903214Z", 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:53.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:53Z" 2024-06-15T09:13:53.361 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:53.362 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.903214_570.1 2024-06-15T09:13:58.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:58.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:13:58.363 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:13:58.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:13:58.409 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:13:58.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:58.461+0000 7f9dc0458a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:58.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:13:58.461+0000 7f9dc0458a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:58.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:13:58.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:58.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:58.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:13:58.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:13:58.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:13:58.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:13:58.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:13:58.487 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:13:58.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.488 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:13:58.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:13:58.488 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:13:58.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:13:58.489 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:13:58Z" 2024-06-15T09:13:58.489 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:13:58.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:03.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:03.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:03.492 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:03.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:03.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:03.590 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:03.589+0000 7fded8904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:03.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:03.593+0000 7fded8904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:03.611 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:03.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:03.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:03.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:03.616 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:03.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:03.616 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:03Z" 2024-06-15T09:14:03.616 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:03.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.979596_577.1 2024-06-15T09:14:08.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:08.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:08.618 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:08.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:08.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:08.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:08.709+0000 7fad400c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:08.709 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:08.709+0000 7fad400c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:08.735 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:08.735 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:08.735 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:08.735 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:08.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:08.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:08.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:08.739 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:08Z" 2024-06-15T09:14:08.740 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:08.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:13.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:13.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:13.743 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:13.786 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:13.786 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:13.829 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:13.829+0000 7fd56548aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:13.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:13.829+0000 7fd56548aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:13.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:13.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:13.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:13.853 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:13.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:13.854 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:13Z" 2024-06-15T09:14:13.854 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:13.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:18.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:18.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:18.856 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:18.899 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:18.899 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:18.951 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:18.949+0000 7f050c418a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:18.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:18.953+0000 7f050c418a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:18.972 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:18.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:18.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:18.975 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:18.976 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:18Z" 2024-06-15T09:14:18.977 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:18.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:23.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:23.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:23.980 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:24.031 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:24.031 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:24.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:24.101+0000 7fd06a4fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:24.102 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:24.101+0000 7fd06a4fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.960893_575.1", 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.960893Z", 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:24Z" 2024-06-15T09:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:24.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.960893_575.1 2024-06-15T09:14:29.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:29.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:29.133 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:29.176 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:29.176 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:29.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:29.221+0000 7fe65cb0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:29.220 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:29.221+0000 7fe65cb0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:29.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:29.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:29.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:29.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.243 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:29.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:29.244 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:29.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:29.244 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:29Z" 2024-06-15T09:14:29.244 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:29.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:34.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:34.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:34.246 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:34.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:34.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:34.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:34.345+0000 7fad9e759a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:34.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:34.345+0000 7fad9e759a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:34.365 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:34.370 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:34Z" 2024-06-15T09:14:34.371 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:34.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:39.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:39.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:39.373 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:39.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:39.416 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:39.459 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:39.457+0000 7f09c69c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:39.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:39.461+0000 7f09c69c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:39.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:39.484 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:39.485 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:39.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:39Z" 2024-06-15T09:14:39.487 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:39.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:44.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:44.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:44.490 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:44.533 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:44.533 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:44.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:44.592+0000 7fbf11c98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:44.592+0000 7fbf11c98a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:44.618 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:44.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:44.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:44.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:44.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:44.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:44Z" 2024-06-15T09:14:44.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:44.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:49.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:49.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:49.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:49.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:49.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:49.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:49.716+0000 7fa7cd6f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:49.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:49.716+0000 7fa7cd6f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:49.736 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:49.737 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:49.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:49.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:49.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:49.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:49.742 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:49.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:49.743 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:49Z" 2024-06-15T09:14:49.743 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:49.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:54.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:54.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:54.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:54.790 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:54.790 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:54.843 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:54.840+0000 7fda4d3bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:54.844 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:54.844+0000 7fda4d3bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:54.863 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:54.864 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:54.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:54.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:54.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:54.868 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:54.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:54.869 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:54.869 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:54.869 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:54Z" 2024-06-15T09:14:54.870 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:54.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:14:59.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:59.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:14:59.872 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:14:59.915 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:14:59.915 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:14:59.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:59.968+0000 7feaf2f15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:59.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:14:59.972+0000 7feaf2f15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:14:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:14:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:14:59Z" 2024-06-15T09:14:59.997 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:14:59.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:04.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:04.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:04.999 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:05.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:05.043 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:05.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:05.092+0000 7f80006caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:05.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:05.092+0000 7f80006caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:05.113 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:05.113 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:05.113 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:05.113 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:05.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:05.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:05.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:05.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:05.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:05.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:05.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:05.117 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:05.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:05.119 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:05Z" 2024-06-15T09:15:05.120 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:05.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:10.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:10.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:10.123 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:10.166 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:10.166 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:10.217 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:10.216+0000 7fb7a05d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:10.218 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:10.216+0000 7fb7a05d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:10.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:10.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.742181_561.1", 2024-06-15T09:15:10.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.742181Z", 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:10.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:10Z" 2024-06-15T09:15:10.242 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:10.243 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.742181_561.1 2024-06-15T09:15:15.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2024-06-15T09:15:15.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:15.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:15.246 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:15.319 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:15.373 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:15:15.402 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718442607.199216_1254.1", 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:10:07.199216Z" 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:15:15.403 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:15:15.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:15:15.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:15.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:15.405 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:15.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:15.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:15.502 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:15.500+0000 7fd14e206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:15.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:15.504+0000 7fd14e206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:15.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:15.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:15.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:15.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:15Z" 2024-06-15T09:15:15.526 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:15.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:20.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:20.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:20.529 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:20.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:20.571 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:20.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:20.612+0000 7f8d5e289a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:20.614 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:20.612+0000 7f8d5e289a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:20.633 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.634 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:20.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:20.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:20.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:20.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:20.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.640 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:20.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:20.641 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:20.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:20.641 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:20Z" 2024-06-15T09:15:20.641 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:20.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:15:25.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:25.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:25.644 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:25.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:25.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:25.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:25.740+0000 7f7f5f657a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:25.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:25.744+0000 7f7f5f657a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:25.763 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:25.763 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:25.763 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:25.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:25.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:25.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:25.767 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:25.768 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:25.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:25.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:25.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:25.772 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:25Z" 2024-06-15T09:15:25.772 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:25.773 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:30.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:30.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:30.775 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:30.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:30.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:30.873 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:30.871+0000 7f252bbb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:30.874 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:30.871+0000 7f252bbb2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:30.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:30.896 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:30.897 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:30.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:30.899 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:30Z" 2024-06-15T09:15:30.900 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:30.900 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:35.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:35.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:35.902 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:35.945 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:35.945 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:36.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:36.003+0000 7ff3d2f1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:36.005 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:36.003+0000 7ff3d2f1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:36.027 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:36.028 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:36.028 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:36.028 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:36.028 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:36.029 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:36.030 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:36.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:36.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:36.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:36.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:36.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:36.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:36.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:36.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:36.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:36.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.038 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:36.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:36.038 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:36.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:36.038 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:36Z" 2024-06-15T09:15:36.039 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:36.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:41.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:41.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:41.043 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:41.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:41.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:41.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:41.127+0000 7f9791aaba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:41.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:41.127+0000 7f9791aaba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:41.152 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:41.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:41.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.156 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:41Z" 2024-06-15T09:15:41.157 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:41.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:15:46.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:46.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:46.159 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:46.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:46.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:46.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:46.251+0000 7f36ea80da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:46.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:46.251+0000 7f36ea80da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:46.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:46.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:46.278 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:46Z" 2024-06-15T09:15:46.278 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:46.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:15:51.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:51.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:51.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:51.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:51.322 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:51.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:51.367+0000 7f3be7bcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:51.369 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:51.367+0000 7f3be7bcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:51.387 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:51.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:51.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:51.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:51.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:51Z" 2024-06-15T09:15:51.392 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:51.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.920756_572.1 2024-06-15T09:15:56.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:56.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:15:56.395 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:15:56.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:15:56.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:15:56.490 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:56.487+0000 7f247110fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:56.491 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:15:56.491+0000 7f247110fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:15:56.515 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:15:56.515 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:15:56.516 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:56.517 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:56.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:15:56.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:15:56Z" 2024-06-15T09:15:56.520 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:15:56.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:01.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:01.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:01.523 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:01.623 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:01.623+0000 7f0ae72d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:01.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:01.623+0000 7f0ae72d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:01.649 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:01.649 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:01.649 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:01.649 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:01.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:01.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:01.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:01.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:01Z" 2024-06-15T09:16:01.654 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:01.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:06.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:06.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:06.656 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:06.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:06.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:06.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:06.739+0000 7f8ed0961a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:06.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:06.743+0000 7f8ed0961a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:06.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:06.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:06.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:06.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:06.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:06.768 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:06Z" 2024-06-15T09:16:06.768 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:06.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:11.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:11.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:11.770 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:11.813 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:11.813 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:11.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:11.859+0000 7fa78a5d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:11.863 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:11.863+0000 7fa78a5d6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:11.882 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:11Z" 2024-06-15T09:16:11.887 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:11.888 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:16:16.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:16.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:16.890 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:16.934 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:16.934 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:16.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:16.975+0000 7f1b022c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:16.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:16.975+0000 7f1b022c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:16.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:16.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:16Z" 2024-06-15T09:16:17.002 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:17.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:22.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:22.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:22.004 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:22.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:22.090+0000 7f815a4b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:22.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:22.090+0000 7f815a4b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:22.112 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:22.112 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:22.112 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:22.112 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:22.112 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:22.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:22.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:22.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:22.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:22Z" 2024-06-15T09:16:22.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:22.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:27.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:27.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:27.119 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:27.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:27.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:27.217 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:27.214+0000 7fd16e775a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:27.218 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:27.214+0000 7fd16e775a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:27.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:27.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:27.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:27.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:27Z" 2024-06-15T09:16:27.243 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:27.245 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:32.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:32.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:32.246 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:32.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:32.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:32.352 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:32.350+0000 7fa4ec235a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:32.353 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:32.350+0000 7fa4ec235a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:32.381 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:32.381 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:32.381 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:32.381 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:32.381 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:32.382 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:32.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:32.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:32.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:32Z" 2024-06-15T09:16:32.386 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:32.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:37.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:37.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:37.388 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:37.432 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:37.432 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:37.490 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:37.486+0000 7f97d27aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:37.491 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:37.490+0000 7f97d27aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:37.555 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:37.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:37.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:37.559 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:37.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:37.560 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:37Z" 2024-06-15T09:16:37.560 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:37.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:42.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:42.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:42.562 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:42.605 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:42.605 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:42.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:42.646+0000 7fe4a289ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:42.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:42.646+0000 7fe4a289ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:42.667 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:42.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:42.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:42.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:42.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:42Z" 2024-06-15T09:16:42.672 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:42.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:47.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:47.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:47.674 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:47.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:47.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:47.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:47.806+0000 7f8af0b6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:47.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:47.806+0000 7f8af0b6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:47.834 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:47.834 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:47.834 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:47.834 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:47.834 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:47.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:47.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.837 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:47.838 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:47Z" 2024-06-15T09:16:47.839 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:47.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:52.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:52.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:52.841 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:52.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:52.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:52.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:52.926+0000 7fde12b25a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:52.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:52.926+0000 7fde12b25a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:52.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:52Z" 2024-06-15T09:16:52.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:52.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:16:57.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:57.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:16:57.955 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:16:58.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:16:58.012 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:16:58.058 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:58.054+0000 7fee4691fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:58.059 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:16:58.058+0000 7fee4691fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:16:58.083 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:16:58.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:58.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:58.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:16:58.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:16:58Z" 2024-06-15T09:16:58.088 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:16:58.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:03.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:03.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:03.091 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:03.134 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:03.134 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:03.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:03.194+0000 7f896af16a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:03.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:03.194+0000 7f896af16a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:03.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:03.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:03.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:03.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:03.221 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:03Z" 2024-06-15T09:17:03.221 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:03.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:17:08.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:08.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:08.223 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:08.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:08.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:08.352 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:08.354+0000 7f8d8fed5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:08.353 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:08.354+0000 7f8d8fed5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:08.377 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:08.377 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:08.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:08.379 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:17:08.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:08.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:08Z" 2024-06-15T09:17:08.382 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:08.383 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:17:13.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:13.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:13.385 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:13.428 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:13.429 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:13.475 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:13.477+0000 7effdeb54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:13.476 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:13.477+0000 7effdeb54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:13.497 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:13.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:13.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:13.501 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:13Z" 2024-06-15T09:17:13.502 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:13.503 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:18.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:18.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:18.504 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:18.549 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:18.549 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:18.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:18.605+0000 7fe57cbbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:18.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:18.609+0000 7fe57cbbba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.631 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.633 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:18.635 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:18Z" 2024-06-15T09:17:18.636 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:18.637 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:23.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:23.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:23.639 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:23.682 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:23.682 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:23.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:23.761+0000 7fea2ac20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:23.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:23.765+0000 7fea2ac20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:23.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:23.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:23.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:23.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:23.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:23Z" 2024-06-15T09:17:23.822 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:23.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:28.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:28.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:28.824 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:28.867 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:28.867 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:28.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:28.929+0000 7f68dadbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:28.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:28.929+0000 7f68dadbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:28.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:28.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:28.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:28.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:28.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:28.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:28Z" 2024-06-15T09:17:28.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:28.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:33.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:33.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:33.956 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:33.999 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:33.999 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:34.076 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:34.077+0000 7f8c987a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:34.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:34.077+0000 7f8c987a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:34.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:34.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:34.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:34.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:34Z" 2024-06-15T09:17:34.101 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:34.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:39.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:39.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:39.103 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:39.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:39.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:39.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:39.201+0000 7f0b898b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:39.202 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:39.201+0000 7f0b898b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:39.220 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:39.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:39.220 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:39.220 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:39.220 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:39.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:39.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:39.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:39.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:39.225 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:39Z" 2024-06-15T09:17:39.225 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:39.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:44.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:44.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:44.227 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:44.270 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:44.270 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:44.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:44.317+0000 7fb5fe712a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:44.316 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:44.317+0000 7fb5fe712a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:44.336 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:44.337 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:44.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:44.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:44.341 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:44Z" 2024-06-15T09:17:44.341 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:44.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:49.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:49.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:49.343 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:49.386 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:49.386 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:49.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:49.437+0000 7f8df1ad3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:49.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:49.441+0000 7f8df1ad3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:49.465 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:49.466 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:49.467 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:49.468 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:49.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:49.470 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:49.471 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:49.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:49.471 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:49Z" 2024-06-15T09:17:49.471 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:49.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:54.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:54.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:54.474 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:54.516 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:54.516 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:54.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:54.569+0000 7fd302ca4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:54.571 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:54.573+0000 7fd302ca4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:54.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:54.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:54.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:54.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:54.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:54Z" 2024-06-15T09:17:54.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:54.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:17:59.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:59.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:17:59.599 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:17:59.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:17:59.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:17:59.704 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:59.705+0000 7f4223d53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:59.705 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:17:59.705+0000 7f4223d53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:17:59.731 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:17:59.731 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:17:59.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:17:59.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:59.734 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:17:59.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:17:59.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:17:59Z" 2024-06-15T09:17:59.737 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:17:59.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:04.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:04.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:04.739 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:04.782 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:04.782 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:04.828 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:04.828+0000 7fa3e6de9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:04.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:04.828+0000 7fa3e6de9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:04.848 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:04.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:04.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:04.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:04.852 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:04.853 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:04Z" 2024-06-15T09:18:04.853 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:04.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:18:09.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:09.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:09.855 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:09.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:09.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:09.951 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:09.952+0000 7f6b8c6dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:09.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:09.952+0000 7f6b8c6dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:09.975 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:09.975 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:09.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:09.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:09.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.979 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:09.980 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:09.980 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:09.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:09.980 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:09Z" 2024-06-15T09:18:09.980 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:09.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:14.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:14.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:14.982 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:15.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:15.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:15.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:15.080+0000 7fbb9796ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:15.081 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:15.080+0000 7fbb9796ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:15.103 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:15.104 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:15.105 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:15.106 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:15Z" 2024-06-15T09:18:15.107 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:15.108 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:18:20.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:20.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:20.110 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:20.151 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:20.151 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:20.217 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:20.216+0000 7f1499d0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:20.217 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:20.216+0000 7f1499d0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:20.237 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:20.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:20.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:20.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:20.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:20.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:20.243 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:20Z" 2024-06-15T09:18:20.243 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:20.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:18:25.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:25.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:25.245 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:25.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:25.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:25.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:25.344+0000 7f1fc43c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:25.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:25.348+0000 7f1fc43c1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:25.366 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:25.366 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:25.366 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:25.366 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:25.366 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:25.367 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:25.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:25.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.369 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:25.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:25.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:25.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:25.372 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:25.373 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:25.374 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:25Z" 2024-06-15T09:18:25.374 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:25.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:30.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:30.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:30.376 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:30.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:30.421 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:30.480 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:30.480+0000 7f1bae09fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:30.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:30.480+0000 7f1bae09fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:30.498 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:30.499 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:30.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:30.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:30Z" 2024-06-15T09:18:30.503 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:30.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:35.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:35.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:35.506 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:35.549 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:35.549 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:35.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:35.600+0000 7f3bf59a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:35.601 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:35.600+0000 7f3bf59a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:35.622 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:35.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:35.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:35.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:35.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:35Z" 2024-06-15T09:18:35.627 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:35.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:18:40.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:40.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:40.630 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:40.679 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:40.679 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:40.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:40.720+0000 7f49e371ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:40.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:40.720+0000 7f49e371ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:40.738 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:40.738 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:40.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:40.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:40.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:40.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:40Z" 2024-06-15T09:18:40.743 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:40.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:45.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:45.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:45.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:45.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:45.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:45.831 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:45.832+0000 7fa75c062a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:45.832 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:45.832+0000 7fa75c062a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:45.850 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:45.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:45.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:45.853 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:45.854 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:45.855 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:45Z" 2024-06-15T09:18:45.856 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:45.856 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:18:50.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:50.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:50.857 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:50.900 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:50.900 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:50.942 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:50.940+0000 7fcd11620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:50.943 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:50.944+0000 7fcd11620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:50.961 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:50.961 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:50.961 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:50.961 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:50.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:50.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:50.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:50.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:50.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:50.967 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:50Z" 2024-06-15T09:18:50.968 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:50.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438537.989943_579.1 2024-06-15T09:18:55.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:55.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:18:55.971 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:18:56.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:18:56.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:18:56.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:56.071+0000 7f1c7e2efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:56.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:18:56.075+0000 7f1c7e2efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:18:56.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:18:56.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:56.100 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:18:56.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:18:56Z" 2024-06-15T09:18:56.102 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:18:56.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.955581_573.1 2024-06-15T09:19:01.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:01.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:01.105 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:01.149 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:01.150 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:01.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:01.191+0000 7f0bff131a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:01.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:01.191+0000 7f0bff131a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.211 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:01Z" 2024-06-15T09:19:01.216 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:01.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.832707_565.1 2024-06-15T09:19:06.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:06.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:06.218 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:06.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:06.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:06.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:06.319+0000 7faf6cf2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:06.323 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:06.323+0000 7faf6cf2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:06.349 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:06.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:06.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:06.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:06.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:06Z" 2024-06-15T09:19:06.355 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:06.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:19:11.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:11.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:11.357 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:11.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:11.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:11.445 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:11.443+0000 7fe2434e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:11.446 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:11.443+0000 7fe2434e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:11.472 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:11.472 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:11.472 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:11.472 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:11.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:11.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.960893_575.1", 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.960893Z", 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:11.475 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:11.476 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:11Z" 2024-06-15T09:19:11.477 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:11.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.960893_575.1 2024-06-15T09:19:16.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:16.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:16.479 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:16.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:16.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:16.569 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:16.567+0000 7f0c40cd6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:16.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:16.567+0000 7f0c40cd6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.907237_571.1", 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.907237Z", 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:16.596 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:16Z" 2024-06-15T09:19:16.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:16.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.907237_571.1 2024-06-15T09:19:21.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:21.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:21.599 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:21.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:21.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:21.706 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:21.703+0000 7fad5c780a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:21.707 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:21.707+0000 7fad5c780a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:21.730 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:21Z" 2024-06-15T09:19:21.735 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:21.736 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:19:26.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:26.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:26.738 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:26.779 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:26.779 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:26.843 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:26.843+0000 7fef15141a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:26.844 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:26.843+0000 7fef15141a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:26.869 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:26.869 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:26.870 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:26.871 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:26.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:26.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:26Z" 2024-06-15T09:19:26.874 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:26.875 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:19:31.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:31.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:31.877 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:31.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:31.920 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:31.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:31.963+0000 7ff489b6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:31.965 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:31.963+0000 7ff489b6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:31.988 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:31.989 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:31.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:31.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:31.992 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:31.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:31.993 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:31Z" 2024-06-15T09:19:31.993 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:31.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438845.799732_672.1 2024-06-15T09:19:36.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:36.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:36.995 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:37.037 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:37.037 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:37.079 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:37.079+0000 7f34bf98da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:37.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:37.079+0000 7f34bf98da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.099 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.100 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:37.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:37.102 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:37.103 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:37.104 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:37.104 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:37Z" 2024-06-15T09:19:37.104 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:37.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.959215_574.1 2024-06-15T09:19:42.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:42.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:42.107 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:42.149 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:42.150 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:42.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:42.191+0000 7f0cd53f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:42.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:42.191+0000 7f0cd53f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:42.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:42.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:42.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:42.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:42.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:42Z" 2024-06-15T09:19:42.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:42.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.959215_574.1 2024-06-15T09:19:47.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:47.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:47.223 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:47.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:47.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:47.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:47.318+0000 7f5a360f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:47.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:47.318+0000 7f5a360f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:47.349 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:47.349 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:47.349 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:47.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:47.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:47.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:47.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.354 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:47.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:47.355 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:47.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:47.355 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:47Z" 2024-06-15T09:19:47.355 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:47.356 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.959215_574.1 2024-06-15T09:19:52.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:52.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:52.357 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:52.401 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:52.401 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:52.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:52.458+0000 7f0e9efdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:52.461 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:52.458+0000 7f0e9efdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.480 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:52.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:52.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:52.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:52Z" 2024-06-15T09:19:52.484 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:52.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.959215_574.1 2024-06-15T09:19:57.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:57.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:19:57.487 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:19:57.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:19:57.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:19:57.573 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:57.570+0000 7fa64033fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:57.574 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:19:57.570+0000 7fa64033fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:57.602 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:57.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:57.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.959215_574.1", 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.959215Z", 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:19:57.605 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:19:57.606 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:19:57Z" 2024-06-15T09:19:57.607 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:19:57.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.959215_574.1 2024-06-15T09:20:02.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:02.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:02.609 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:02.651 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:02.651 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:02.707 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:02.702+0000 7fcd4f727a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:02.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:02.706+0000 7fcd4f727a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:02.729 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:02.730 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:02.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:02.734 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:02.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:02.734 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:02Z" 2024-06-15T09:20:02.734 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:02.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:07.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:07.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:07.736 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:07.779 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:07.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:07.838 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:07.834+0000 7f60cb455a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:07.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:07.834+0000 7f60cb455a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:07.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:07.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:07.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:07.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:07Z" 2024-06-15T09:20:07.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:07.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:12.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:12.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:12.867 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:12.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:12.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:12.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:12.958+0000 7f59753f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:12.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:12.958+0000 7f59753f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:12.986 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:12.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:12.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:12.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:12.990 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:12.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:12.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:12Z" 2024-06-15T09:20:12.993 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:12.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:17.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:17.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:17.996 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:18.040 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:18.040 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:18.087 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:18.082+0000 7f084723aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:18.087 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:18.086+0000 7f084723aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:18.106 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:18.106 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:18.106 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:18.106 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:18.106 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:18.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:18.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:18.109 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:18.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:18Z" 2024-06-15T09:20:18.111 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:18.112 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718442607.199216_1254.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:23.113 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2024-06-15T09:20:23.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:23.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:23.169 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:23.213 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:23.279 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:20:23.365 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443223.165103_1312.1", 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:20:23.165103Z" 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:20:23.366 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:20:23.367 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:20:23.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:23.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:23.368 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:23.410 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:23.411 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:23.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:23.478+0000 7ff0cd961a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:23.482 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:23.478+0000 7ff0cd961a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:23.501 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:23.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:23.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:23.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:23.506 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:23Z" 2024-06-15T09:20:23.507 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:23.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:28.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:28.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:28.509 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:28.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:28.552 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:28.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:28.614+0000 7f68a14c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:28.614 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:28.614+0000 7f68a14c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:28.640 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:28.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:28.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:28.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:28.644 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:28Z" 2024-06-15T09:20:28.645 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:28.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:33.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:33.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:33.648 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:33.690 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:33.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:33.770+0000 7ff1de053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:33.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:33.770+0000 7ff1de053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:33.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:33.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:33.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.800 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:33.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:33.801 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:33.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:33.801 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:33Z" 2024-06-15T09:20:33.801 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:33.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:38.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:38.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:38.803 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:38.847 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:38.847 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:38.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:38.901+0000 7fd77c80fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:38.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:38.901+0000 7fd77c80fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.922 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:38.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:38.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:38.926 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:38.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:38.927 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:38Z" 2024-06-15T09:20:38.927 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:38.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:43.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:43.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:43.929 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:43.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:43.977 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:44.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:44.017+0000 7fa46ae4fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:44.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:44.017+0000 7fa46ae4fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:44.036 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:44.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:44.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:20:44.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:44.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:44Z" 2024-06-15T09:20:44.041 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:44.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:20:49.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:49.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:49.043 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:49.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:49.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:49.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:49.133+0000 7fb814e17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:49.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:49.133+0000 7fb814e17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:49.152 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:49Z" 2024-06-15T09:20:49.157 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:49.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:54.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:54.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:54.158 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:54.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:54.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:54.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:54.245+0000 7f35d2769a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:54.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:54.245+0000 7f35d2769a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:54.263 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:54.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:54.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:54Z" 2024-06-15T09:20:54.268 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:54.269 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:20:59.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:59.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:20:59.270 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:20:59.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:20:59.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:20:59.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:59.369+0000 7f61e6a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:59.369 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:20:59.369+0000 7f61e6a35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:20:59.387 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:20:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:20:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:20:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:20:59.387 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:20:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:20:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:20:59Z" 2024-06-15T09:20:59.392 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:20:59.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:04.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:04.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:04.394 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:04.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:04.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:04.480 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:04.481+0000 7f120fcdea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:04.481 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:04.481+0000 7f120fcdea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:04.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:04.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:04.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:04.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:04Z" 2024-06-15T09:21:04.505 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:04.506 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:09.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:09.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:09.508 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:09.550 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:09.550 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:09.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:09.593+0000 7fb542b1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:09.594 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:09.593+0000 7fb542b1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:09.616 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:09.616 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:09.616 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:09.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:09.618 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.920756_572.1", 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.920756Z", 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:09Z" 2024-06-15T09:21:09.621 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:09.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.920756_572.1 2024-06-15T09:21:14.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:14.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:14.623 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:14.666 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:14.666 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:14.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:14.717+0000 7ff43e22fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:14.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:14.717+0000 7ff43e22fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:14.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:14.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:14Z" 2024-06-15T09:21:14.742 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:14.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:19.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:19.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:19.744 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:19.787 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:19.787 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:19.828 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:19.829+0000 7f686a1f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:19.829 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:19.829+0000 7f686a1f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:19.848 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:19.848 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:19.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:19.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:19.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:19Z" 2024-06-15T09:21:19.853 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:19.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:24.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:24.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:24.856 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:24.898 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:24.898 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:24.942 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:24.941+0000 7f92e52b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:24.943 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:24.945+0000 7f92e52b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:24.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:24.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:24.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:24Z" 2024-06-15T09:21:24.967 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:24.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:29.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:29.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:29.970 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:30.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:30.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:30.063 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:30.060+0000 7f79b1884a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:30.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:30.064+0000 7f79b1884a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:30.082 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:30.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:30.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:30.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:30.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:30Z" 2024-06-15T09:21:30.087 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:30.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:35.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:35.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:35.089 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:35.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:35.133 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:35.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:35.180+0000 7fd4cbfc1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:35.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:35.180+0000 7fd4cbfc1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:35.205 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:35.205 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:35.205 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:35.205 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:35.206 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:35.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:35.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:35.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:35Z" 2024-06-15T09:21:35.210 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:35.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:40.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:40.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:40.213 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:40.254 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:40.254 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:40.313 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:40.312+0000 7f2b4dfbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:40.314 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:40.312+0000 7f2b4dfbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:40.336 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:40.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:40.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:40.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:40.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:40Z" 2024-06-15T09:21:40.341 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:40.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:45.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:45.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:45.344 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:45.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:45.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:45.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:45.440+0000 7fd0a057fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:45.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:45.440+0000 7fd0a057fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:45Z" 2024-06-15T09:21:45.466 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:45.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:21:50.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:50.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:50.469 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:50.512 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:50.512 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:50.556 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:50.556+0000 7f619cca8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:50.556 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:50.556+0000 7f619cca8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:50.581 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:50.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:50.585 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:50Z" 2024-06-15T09:21:50.586 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:50.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:21:55.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:55.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:21:55.588 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:21:55.631 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:21:55.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:21:55.679 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:55.680+0000 7f0eb29b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:55.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:21:55.680+0000 7f0eb29b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:21:55.698 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:21:55.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:55.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:21:55.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:21:55.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:21:55.704 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:21:55.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:21:55.704 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:21:55Z" 2024-06-15T09:21:55.704 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:21:55.705 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:00.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:00.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:00.706 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:00.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:00.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:00.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:00.812+0000 7f105fbc3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:00.814 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:00.812+0000 7f105fbc3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:00.831 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:00Z" 2024-06-15T09:22:00.836 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:00.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:05.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:05.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:05.839 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:05.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:05.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:05.939 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:05.940+0000 7f901bf54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:05.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:05.940+0000 7f901bf54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:05.966 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:05.966 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:05.966 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:05.966 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:05.966 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:05.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:05.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.970 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:05.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:05.971 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:05.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:05.971 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:05Z" 2024-06-15T09:22:05.971 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:05.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:10.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:10.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:10.973 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:11.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:11.016 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:11.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:11.072+0000 7f82b74bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:11.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:11.072+0000 7f82b74bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:11.096 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:11.096 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:11.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:11.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:11.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:11Z" 2024-06-15T09:22:11.102 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:11.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:16.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:16.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:16.104 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:16.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:16.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:16.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:16.192+0000 7fd003602a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:16.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:16.196+0000 7fd003602a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:16.221 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:16.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:16Z" 2024-06-15T09:22:16.227 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:16.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:21.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:21.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:21.230 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:21.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:21.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:21.317 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:21.315+0000 7f950c9c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:21.318 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:21.315+0000 7f950c9c4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:21.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:21.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:21.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:21.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:21.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:21.342 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:21Z" 2024-06-15T09:22:21.342 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:21.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:22:26.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:26.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:26.344 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:26.391 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:26.391 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:26.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:26.439+0000 7f9863c3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:26.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:26.439+0000 7f9863c3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:26.462 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:26.462 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:26.465 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:26.467 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:26.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:26.469 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:26.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:26.470 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:26Z" 2024-06-15T09:22:26.470 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:26.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:31.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:31.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:31.472 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:31.515 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:31.516 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:31.572 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:31.571+0000 7faac7752a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:31.573 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:31.571+0000 7faac7752a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:31.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:31.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:31.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:31.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:31.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:31.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:31.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:31Z" 2024-06-15T09:22:31.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:31.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:22:36.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:36.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:36.599 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:36.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:36.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:36.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:36.683+0000 7f40f4a0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:36.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:36.683+0000 7f40f4a0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:36.707 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:36.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:36.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:36.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:36.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:36.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:36.712 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:36Z" 2024-06-15T09:22:36.713 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:36.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:41.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:41.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:41.716 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:41.759 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:41.759 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:41.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:41.807+0000 7fca2abcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:41.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:41.811+0000 7fca2abcea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:41.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:41.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:41.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:41.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:41.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:41Z" 2024-06-15T09:22:41.835 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:41.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:46.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:46.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:46.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:46.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:46.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:46.944 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:46.943+0000 7feb46751a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:46.945 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:46.943+0000 7feb46751a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:46.966 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:46.966 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:46.966 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:46.966 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:46.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:46.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:22:46.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:46.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:46Z" 2024-06-15T09:22:46.971 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:46.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:22:51.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:51.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:51.973 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:52.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:52.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:52.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:52.067+0000 7f4097f59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:52.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:52.067+0000 7f4097f59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:52.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:52.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:52.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:52.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:52.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:52Z" 2024-06-15T09:22:52.094 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:52.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:22:57.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:57.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:22:57.097 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:22:57.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:22:57.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:22:57.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:57.195+0000 7f46b04b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:57.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:22:57.195+0000 7f46b04b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:22:57.222 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:22:57.222 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:22:57.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:22:57.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:57.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:22:57.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:22:57Z" 2024-06-15T09:22:57.227 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:22:57.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:02.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:02.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:02.230 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:02.271 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:02.271 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:02.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:02.319+0000 7fe0f6d59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:02.323 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:02.319+0000 7fe0f6d59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:02.345 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:02Z" 2024-06-15T09:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:02.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:07.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:07.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:07.352 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:07.394 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:07.395 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:07.438 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:07.435+0000 7fbfc2594a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:07.439 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:07.439+0000 7fbfc2594a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:07.459 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:07.459 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:07.459 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:07.459 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:07.460 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:07.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:07.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:07Z" 2024-06-15T09:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:07.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:12.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:12.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:12.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:12.510 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:12.510 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:12.571 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:12.570+0000 7f2ab71c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:12.572 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:12.570+0000 7f2ab71c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:12.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:12.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:12.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:12.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:12.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:12.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:12.598 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:12Z" 2024-06-15T09:23:12.598 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:12.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:17.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:17.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:17.600 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:17.644 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:17.644 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:17.698 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:17.694+0000 7fac29b51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:17.699 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:17.698+0000 7fac29b51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:17Z" 2024-06-15T09:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:17.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:22.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:22.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:22.725 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:22.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:22.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:22.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:22.806+0000 7fc1284bca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:22.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:22.810+0000 7fc1284bca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:22.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:22.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:22.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:22.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:22.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:22Z" 2024-06-15T09:23:22.837 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:22.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:23:27.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:27.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:27.839 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:27.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:27.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:27.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:27.922+0000 7f985a382a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:27.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:27.926+0000 7f985a382a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:27.954 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:27Z" 2024-06-15T09:23:27.955 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:27.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:32.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:32.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:32.958 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:33.042 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:33.038+0000 7fbd67638a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:33.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:33.042+0000 7fbd67638a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:33.065 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:33.065 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:33.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:33.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:33.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:33.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:33Z" 2024-06-15T09:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:33.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:38.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:38.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:38.074 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:38.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:38.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:38.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:38.226+0000 7fdbfd491a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:38.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:38.226+0000 7fdbfd491a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:38.252 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:38.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.254 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:38.255 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:38.256 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:38.257 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:38.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:38.258 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:38Z" 2024-06-15T09:23:38.258 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:38.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:23:43.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:43.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:43.261 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:43.304 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:43.304 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:43.347 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:43.346+0000 7fe166c0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:43.348 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:43.346+0000 7fe166c0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:43.367 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:43.367 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:43.367 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:43.367 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:43.367 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:43.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:43.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:43.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:43.371 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:43Z" 2024-06-15T09:23:43.372 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:43.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:48.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:48.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:48.375 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:48.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:48.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:48.466 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:48.462+0000 7f72bcc37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:48.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:48.466+0000 7f72bcc37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:48.509 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:48.509 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:48.509 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:48.509 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:48.509 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:48.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:48.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:48.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:48.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:48Z" 2024-06-15T09:23:48.514 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:48.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:23:53.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:53.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:53.518 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:53.561 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:53.561 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:53.606 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:53.606+0000 7f14e334aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:53.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:53.610+0000 7f14e334aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.626 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:53.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:53.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:53.630 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:53.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:53.631 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:53Z" 2024-06-15T09:23:53.631 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:53.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:23:58.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:58.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:23:58.634 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:23:58.678 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:23:58.678 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:23:58.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:58.734+0000 7f294ab6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:58.735 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:23:58.738+0000 7f294ab6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:23:58.761 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:23:58.761 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:23:58.761 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:23:58.762 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:23:58.762 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:23:58.762 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:23:58.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:58.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:58.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:58.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:58.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:23:58.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:58.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:58.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:58.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:23:58.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:58.767 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:23:58.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:23:58.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:23:58.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:23:58.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:23:58.772 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:23:58.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:23:58.772 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:23:58Z" 2024-06-15T09:23:58.772 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:23:58.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:24:03.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:03.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:03.775 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:03.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:03.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:03.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:03.881+0000 7f4cd6013a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:03.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:03.881+0000 7f4cd6013a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.908 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.910 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.960893_575.1", 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.960893Z", 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.912 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:03.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:03.913 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:03.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:03.913 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:03Z" 2024-06-15T09:24:03.913 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:03.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.960893_575.1 2024-06-15T09:24:08.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:08.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:08.915 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:08.961 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:08.961 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:09.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:09.021+0000 7f132b536a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:09.023 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:09.025+0000 7f132b536a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:09.048 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:09.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:09.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:09.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:09.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:09Z" 2024-06-15T09:24:09.053 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:09.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:14.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:14.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:14.056 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:14.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:14.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:14.166 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:14.165+0000 7fa0191f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:14.167 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:14.165+0000 7fa0191f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:14.184 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:14.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:14.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:14.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:14.189 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:14Z" 2024-06-15T09:24:14.189 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:14.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:19.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:19.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:19.191 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:19.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:19.236 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:19.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:19.281+0000 7f356802ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:19.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:19.281+0000 7f356802ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:19.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:19.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:19.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:19.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:19Z" 2024-06-15T09:24:19.308 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:19.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:24.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:24.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:24.310 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:24.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:24.367 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:24.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:24.413+0000 7fa7edd60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:24.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:24.413+0000 7fa7edd60a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:24.431 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:24.432 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:24.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:24.434 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:24.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:24.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:24.437 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:24Z" 2024-06-15T09:24:24.437 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:24.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:24:29.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:29.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:29.439 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:29.483 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:29.483 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:29.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:29.537+0000 7f03cb785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:29.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:29.537+0000 7f03cb785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:29.556 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:29.556 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:29.556 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:29.556 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:29.556 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:29.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:29.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:29.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:29.561 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:29.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:29.561 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:29Z" 2024-06-15T09:24:29.561 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:29.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:34.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:34.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:34.562 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:34.605 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:34.605 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:34.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:34.653+0000 7f659638ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:34.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:34.657+0000 7f659638ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:34.678 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:34.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:34.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:34.682 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:34Z" 2024-06-15T09:24:34.683 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:34.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:39.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:39.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:39.685 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:39.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:39.728 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:39.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:39.781+0000 7f76c49a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:39.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:39.785+0000 7f76c49a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:39.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:39.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:39.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:39.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:39Z" 2024-06-15T09:24:39.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:39.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:44.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:44.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:44.817 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:44.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:44.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:44.919 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:44.921+0000 7f4ad4a46a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:44.920 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:44.921+0000 7f4ad4a46a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:44.943 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:44.943 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:44.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:44.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:44.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:44.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:44Z" 2024-06-15T09:24:44.948 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:44.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:24:49.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:49.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:49.951 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:49.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:49.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:50.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:50.061+0000 7f883f17fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:50.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:50.061+0000 7f883f17fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:50.092 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:50Z" 2024-06-15T09:24:50.097 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:50.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.979596_577.1 2024-06-15T09:24:55.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:55.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:24:55.099 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:24:55.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:24:55.142 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:24:55.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:55.192+0000 7f414028da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:55.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:24:55.196+0000 7f414028da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:55.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:24:55.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:24:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:24:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:24:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:24:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:24:55Z" 2024-06-15T09:24:55.221 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:24:55.222 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:25:00.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:00.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:00.223 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:00.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:00.269 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:00.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:00.344+0000 7f170c785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:00.346 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:00.344+0000 7f170c785a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:00.365 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:00.366 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:00.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:00.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.369 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:00.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:00.370 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:00.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:00.370 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:00Z" 2024-06-15T09:25:00.370 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:00.371 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.832707_565.1 2024-06-15T09:25:05.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:05.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:05.372 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:05.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:05.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:05.459 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:05.460+0000 7f6e26d9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:05.460 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:05.460+0000 7f6e26d9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:05.479 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:05.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:05.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:05.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:05.483 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:05Z" 2024-06-15T09:25:05.484 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:05.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438845.915314_676.1 2024-06-15T09:25:10.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:10.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:10.486 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:10.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:10.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:10.573 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:10.572+0000 7fa612b1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:10.574 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:10.572+0000 7fa612b1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:10.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:10.592 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:10.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:10.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:10.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:25:10.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:10.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:10Z" 2024-06-15T09:25:10.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:10.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:25:15.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:15.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:15.600 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:15.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:15.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:15.690 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:15.688+0000 7f735d58ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:15.691 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:15.692+0000 7f735d58ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:15.710 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:15.711 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:15.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:15.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:15.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:15Z" 2024-06-15T09:25:15.715 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:15.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.759965_563.1 2024-06-15T09:25:20.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:20.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:20.718 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:20.774 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:20.774 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:20.848 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:20.848+0000 7fd0c3d03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:20.848 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:20.848+0000 7fd0c3d03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:20.867 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:20.867 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:20.868 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:20.869 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:20.870 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:20.871 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:20Z" 2024-06-15T09:25:20.872 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:20.876 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.895427_569.1 2024-06-15T09:25:25.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:25.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:25.878 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:25.922 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:25.922 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:25.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:25.964+0000 7f8a1f9d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:25.965 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:25.964+0000 7f8a1f9d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.985 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.986 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:25.987 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:25.988 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:25Z" 2024-06-15T09:25:25.989 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:25.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443223.165103_1312.1 target=1_1718438536.955581_573.1 2024-06-15T09:25:30.992 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2024-06-15T09:25:31.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:31.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:31.026 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:31.119 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:25:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "", 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "0.000000" 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443531.023632_1339.1", 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:25:31.023632Z" 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:40:20.458611Z" 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:25:31.147 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:25:31.148 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:25:31.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:31.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:31.149 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:31.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:31.189 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:31.242 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:31.240+0000 7f9e03e69a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:31.243 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:31.244+0000 7f9e03e69a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:31.262 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:31.262 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:31.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:31.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.960893_575.1", 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.960893Z", 2024-06-15T09:25:31.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:31.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:31Z" 2024-06-15T09:25:31.267 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:31.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.960893_575.1 2024-06-15T09:25:36.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:36.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:36.269 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:36.313 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:36.313 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:36.376 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:36.376+0000 7f23f86dba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:36.377 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:36.376+0000 7f23f86dba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:36.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:36.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:36.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:36.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:36.416 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:36.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:36.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:36Z" 2024-06-15T09:25:36.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:36.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:25:41.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:41.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:41.418 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:41.461 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:41.461 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:41.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:41.520+0000 7f5b44ceca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:41.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:41.520+0000 7f5b44ceca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:41.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:41.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.759965_563.1", 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.759965Z", 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:41.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:41Z" 2024-06-15T09:25:41.544 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:41.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.759965_563.1 2024-06-15T09:25:46.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:46.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:46.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:46.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:46.594 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:46.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:46.647+0000 7fd345f35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:46.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:46.647+0000 7fd345f35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:46.669 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:46.669 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:46.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:46.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:46.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:46.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:46Z" 2024-06-15T09:25:46.674 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:46.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:25:51.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:51.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:51.677 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:51.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:51.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:51.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:51.763+0000 7f5e773afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:51.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:51.763+0000 7f5e773afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:51.785 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:51.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:51.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:51.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:51.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:51.790 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:51Z" 2024-06-15T09:25:51.791 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:51.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:25:56.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:56.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:25:56.792 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:25:56.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:25:56.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:25:56.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:56.891+0000 7fde84427a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:56.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:25:56.891+0000 7fde84427a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:25:56.914 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:56.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:25:56.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:56.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:25:56.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:25:56.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:56.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:25:56.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:25:56Z" 2024-06-15T09:25:56.920 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:25:56.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:01.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:01.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:01.923 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:01.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:01.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:02.006 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:02.003+0000 7ff34d12ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:02.008 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:02.007+0000 7ff34d12ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:02.041 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.042 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:02.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:02.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:26:02.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:02.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:02.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:02Z" 2024-06-15T09:26:02.048 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:02.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:26:07.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:07.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:07.050 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:07.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:07.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:07.148 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:07.147+0000 7fc4ed9f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:07.149 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:07.147+0000 7fc4ed9f1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:07.174 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:07.174 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:07.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:07.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:07.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:07.178 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:07Z" 2024-06-15T09:26:07.179 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:07.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:12.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:12.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:12.181 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:12.223 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:12.223 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:12.272 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:12.271+0000 7f5c4eaf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:12.273 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:12.271+0000 7f5c4eaf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:12.299 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:12.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:12.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:12.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:12.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:12Z" 2024-06-15T09:26:12.304 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:12.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:26:17.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:17.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:17.307 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:17.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:17.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:17.412 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:17.411+0000 7f4854b9ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:17.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:17.411+0000 7f4854b9ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:17.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:17.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:17.438 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:17.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:17.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:17.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:17Z" 2024-06-15T09:26:17.442 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:17.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:22.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:22.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:22.444 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:22.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:22.487 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:22.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:22.551+0000 7f62cc1f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:22.555 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:22.555+0000 7f62cc1f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:22.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:22.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:22.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:22.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:22.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:22Z" 2024-06-15T09:26:22.582 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:22.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:26:27.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:27.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:27.585 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:27.627 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:27.627 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:27.668 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:27.667+0000 7f3ff28d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:27.669 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:27.667+0000 7f3ff28d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:27.708 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:27.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:27.710 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:27.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:27.712 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:27Z" 2024-06-15T09:26:27.713 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:27.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:32.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:32.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:32.716 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:32.761 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:32.761 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:32.808 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:32.807+0000 7f29c4ecaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:32.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:32.807+0000 7f29c4ecaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:32.828 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:32.828 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:32.828 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:32.829 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:32.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:32.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:32.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:32Z" 2024-06-15T09:26:32.833 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:32.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:26:37.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:37.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:37.835 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:37.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:37.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:37.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:37.922+0000 7f2a60a1ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:37.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:37.926+0000 7f2a60a1ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:37.946 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:37.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:37.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:26:37.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:37.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:37Z" 2024-06-15T09:26:37.951 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:37.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:26:42.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:42.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:42.954 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:42.997 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:42.997 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:43.049 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:43.046+0000 7f90bd1e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:43.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:43.046+0000 7f90bd1e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:43.067 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:43.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:43.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:43.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:43Z" 2024-06-15T09:26:43.072 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:43.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:26:48.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:48.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:48.074 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:48.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:48.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:48.170 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:48.166+0000 7fa380ca9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:48.171 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:48.170+0000 7fa380ca9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:48.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:48.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:48.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:48.194 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:48.195 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:48.196 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:48.196 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:48Z" 2024-06-15T09:26:48.196 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:48.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:53.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:53.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:53.197 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:53.241 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:53.241 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:53.296 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:53.294+0000 7ff77ed35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:53.297 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:53.294+0000 7ff77ed35a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:53.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:53.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:53.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:53.327 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:53Z" 2024-06-15T09:26:53.328 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:53.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:26:58.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:58.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:26:58.330 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:26:58.375 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:26:58.375 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:26:58.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:58.422+0000 7f2dcf178a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:58.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:26:58.426+0000 7f2dcf178a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:58.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:58.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:26:58.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.964810_576.1", 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.964810Z", 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:26:58.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:26:58.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:26:58.454 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:26:58Z" 2024-06-15T09:26:58.454 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:26:58.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.964810_576.1 2024-06-15T09:27:03.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:03.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:03.456 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:03.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:03.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:03.565 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:03.562+0000 7f4a27819a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:03.566 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:03.562+0000 7f4a27819a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:03.587 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:03.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:03.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:03.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:03.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:03Z" 2024-06-15T09:27:03.592 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:03.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:08.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:08.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:08.595 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:08.638 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:08.638 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:08.700 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:08.698+0000 7f250149aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:08.701 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:08.698+0000 7f250149aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:08.722 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:08.722 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:08.722 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:08.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:08.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:08.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:08.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:08Z" 2024-06-15T09:27:08.727 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:08.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:27:13.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:13.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:13.730 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:13.773 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:13.773 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:13.838 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:13.834+0000 7f2316a39a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:13.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:13.838+0000 7f2316a39a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:13.880 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:13.880 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:13.880 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:13.880 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:13.881 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:13.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:13.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:13Z" 2024-06-15T09:27:13.885 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:13.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:18.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:18.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:18.888 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:18.932 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:18.932 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:18.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:18.978+0000 7fc7defdca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:18.979 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:18.982+0000 7fc7defdca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:19.000 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:19.000 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:19.000 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:19.000 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:19.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:19.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:19.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:19.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:18Z" 2024-06-15T09:27:19.005 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:19.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:24.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:24.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:24.008 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:24.051 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:24.051 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:24.102 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:24.102+0000 7f213a335a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:24.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:24.106+0000 7f213a335a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:24.125 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:24.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:24.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:24Z" 2024-06-15T09:27:24.130 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:24.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:29.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:29.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:29.133 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:29.177 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:29.177 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:29.224 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:29.225+0000 7f0710a87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:29.225 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:29.225+0000 7f0710a87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:29.244 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:29.245 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:29.246 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:29.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:29.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:29.249 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:29.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:29.250 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:29Z" 2024-06-15T09:27:29.250 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:29.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:34.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:34.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:34.252 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:34.293 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:34.294 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:34.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:34.345+0000 7fb9c3fbda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:34.345 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:34.345+0000 7fb9c3fbda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:34.363 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:34.364 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:34.365 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:34.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:34.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:34.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:34.369 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:34Z" 2024-06-15T09:27:34.369 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:34.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:39.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:39.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:39.371 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:39.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:39.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:39.467 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:39.469+0000 7fe624397a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:39.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:39.469+0000 7fe624397a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:39.489 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:39.489 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:39.489 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:39.489 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:39.489 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:39.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:39.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:39.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:39Z" 2024-06-15T09:27:39.494 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:39.495 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:44.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:44.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:44.497 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:44.541 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:44.541 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:44.603 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:44.605+0000 7f1cffa06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:44.604 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:44.605+0000 7f1cffa06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:44.625 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:44.625 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:44.625 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:44.625 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:44.625 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:44.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:44.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:44.628 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:44.629 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:44.630 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:44.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:44.631 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:44Z" 2024-06-15T09:27:44.631 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:44.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:49.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:49.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:49.633 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:49.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:49.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:49.723 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:49.725+0000 7f309ca7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:49.724 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:49.725+0000 7f309ca7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:49.743 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:49.743 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.744 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:49.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.746 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:49.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:49.748 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:49.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:49.749 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:49Z" 2024-06-15T09:27:49.749 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:49.750 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:54.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:54.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:54.751 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:54.794 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:54.795 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:54.837 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:54.837+0000 7f4be284ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:54.838 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:54.837+0000 7f4be284ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:54.856 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:54.856 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:54.856 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:54.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:54.858 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:54.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:54.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:54Z" 2024-06-15T09:27:54.861 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:54.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:27:59.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:59.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:27:59.864 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:27:59.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:27:59.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:27:59.967 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:59.969+0000 7f3cda12ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:59.967 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:27:59.969+0000 7f3cda12ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.992 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:59.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:27:59.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:27:59Z" 2024-06-15T09:27:59.997 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:27:59.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:04.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:04.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:04.999 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:05.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:05.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:05.104 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:05.105+0000 7f9fa03f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:05.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:05.105+0000 7f9fa03f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:05.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:05.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:05.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:05.152 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:05.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:05.153 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:05Z" 2024-06-15T09:28:05.153 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:05.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:10.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:10.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:10.155 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:10.198 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:10.199 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:10.248 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:10.249+0000 7fc3190b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:10.248 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:10.249+0000 7fc3190b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:10.271 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:10.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.907237_571.1", 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.907237Z", 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:10Z" 2024-06-15T09:28:10.277 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:10.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.907237_571.1 2024-06-15T09:28:15.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:15.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:15.279 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:15.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:15.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:15.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:15.373+0000 7f8d3fe65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:15.373 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:15.373+0000 7f8d3fe65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:15.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:15.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:15.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:15.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:15.396 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:15Z" 2024-06-15T09:28:15.397 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:15.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:20.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:20.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:20.400 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:20.441 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:20.441 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:20.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:20.536+0000 7f56e0bb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:20.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:20.540+0000 7f56e0bb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:20.568 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:20.568 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:20.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:20.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.907237_571.1", 2024-06-15T09:28:20.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.907237Z", 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:20.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:20Z" 2024-06-15T09:28:20.573 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:20.574 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.907237_571.1 2024-06-15T09:28:25.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:25.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:25.577 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:25.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:25.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:25.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:25.664+0000 7fa34065ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:25.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:25.664+0000 7fa34065ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:25.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:25.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:25.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:25.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:25.688 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:25Z" 2024-06-15T09:28:25.688 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:25.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.955581_573.1 2024-06-15T09:28:30.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:30.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:30.690 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:30.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:30.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:30.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:30.812+0000 7f9f299cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:30.812 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:30.812+0000 7f9f299cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:30.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:30.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:30Z" 2024-06-15T09:28:30.835 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:30.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:35.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:35.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:35.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:35.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:35.929 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:35.928+0000 7fa4dbd47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:35.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:35.928+0000 7fa4dbd47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:35.950 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:28:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:35.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:35Z" 2024-06-15T09:28:35.955 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:35.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:28:40.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:40.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:40.958 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:41.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:41.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:41.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:41.048+0000 7f45dbb56a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:41.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:41.052+0000 7f45dbb56a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:41.072 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:41.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:41.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:41.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:41.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:41Z" 2024-06-15T09:28:41.077 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:41.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:46.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:46.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:46.079 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:46.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:46.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:46.167 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:46.168+0000 7f4f1c651a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:46.168 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:46.168+0000 7f4f1c651a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:46.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:46.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:46.190 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:46.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:46.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:46.193 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:46Z" 2024-06-15T09:28:46.194 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:46.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:28:51.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:51.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:51.196 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:51.240 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:51.240 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:51.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:51.296+0000 7f5fadb2aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:51.298 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:51.300+0000 7f5fadb2aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:51.318 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:51.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:51.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:28:51.321 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:51.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:51.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:51.324 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:51.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:51.324 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:51Z" 2024-06-15T09:28:51.324 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:51.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:28:56.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:56.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:28:56.326 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:28:56.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:28:56.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:28:56.415 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:56.416+0000 7f0485b65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:56.416 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:28:56.416+0000 7f0485b65a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:28:56.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:56.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:56.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:28:56.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:28:56.443 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:28:56Z" 2024-06-15T09:28:56.444 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:28:56.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.955581_573.1 2024-06-15T09:29:01.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:01.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:01.445 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:01.489 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:01.489 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:01.532 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:01.532+0000 7f4eac1d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:01.533 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:01.532+0000 7f4eac1d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:01.552 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:01.552 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:01.552 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:01.553 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:01.554 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:01.555 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:01Z" 2024-06-15T09:29:01.557 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:01.558 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:06.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:06.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:06.559 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:06.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:06.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:06.645 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:06.644+0000 7f79c87afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:06.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:06.644+0000 7f79c87afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:06.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:06.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:06.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:06.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:06.670 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:06Z" 2024-06-15T09:29:06.670 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:06.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:11.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:11.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:11.672 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:11.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:11.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:11.769 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:11.767+0000 7f84850eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:11.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:11.767+0000 7f84850eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:11.791 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:11.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:11Z" 2024-06-15T09:29:11.796 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:11.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:16.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:16.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:16.798 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:16.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:16.849 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:16.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:16.895+0000 7f79d748ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:16.898 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:16.895+0000 7f79d748ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:16.917 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:16.917 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:16.917 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:16.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:16.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:16.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:16.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:16Z" 2024-06-15T09:29:16.922 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:16.923 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:21.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:21.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:21.925 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:21.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:21.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:22.012 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:22.011+0000 7f8e6337ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:22.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:22.011+0000 7f8e6337ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:22.031 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:22.031 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:22.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:22.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:22.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:22.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:22Z" 2024-06-15T09:29:22.037 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:22.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:29:27.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:27.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:27.039 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:27.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:27.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:27.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:27.135+0000 7f26e157aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:27.139 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:27.139+0000 7f26e157aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:27.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:27Z" 2024-06-15T09:29:27.163 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:27.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:32.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:32.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:32.165 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:32.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:32.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:32.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:32.259+0000 7f78aa9c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:32.261 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:32.259+0000 7f78aa9c0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:32.284 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:32.284 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:32.284 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:32.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:32.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:32Z" 2024-06-15T09:29:32.289 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:32.290 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:37.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:37.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:37.292 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:37.338 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:37.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:37.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:37.387+0000 7f1a4e0ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:37.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:37.387+0000 7f1a4e0ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:37.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:37.416 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:37.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:37.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:37Z" 2024-06-15T09:29:37.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:37.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:29:42.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:42.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:42.418 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:42.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:42.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:42.506 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:42.503+0000 7f0f6aa87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:42.507 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:42.507+0000 7f0f6aa87a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:42.527 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:42.527 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:42.527 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:42.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:42.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.832707_565.1", 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.832707Z", 2024-06-15T09:29:42.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:42Z" 2024-06-15T09:29:42.532 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:42.533 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.832707_565.1 2024-06-15T09:29:47.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:47.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:47.534 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:47.577 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:47.577 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:47.622 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:47.619+0000 7f9c0da3ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:47.623 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:47.623+0000 7f9c0da3ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:47.648 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:47.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:47.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:47.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:47Z" 2024-06-15T09:29:47.654 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:47.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:52.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:52.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:52.656 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:52.699 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:52.699 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:52.751 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:52.751+0000 7feadcf78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:52.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:52.751+0000 7feadcf78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:52.771 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:52.771 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:52.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:52.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:52.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:52.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:52Z" 2024-06-15T09:29:52.776 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:52.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:29:57.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:57.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:29:57.782 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:29:57.825 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:29:57.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:29:57.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:57.883+0000 7f3edce05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:57.884 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:29:57.883+0000 7f3edce05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.925 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:57.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:29:57.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.929 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:29:57.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:29:57.930 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:29:57.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:29:57.930 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:29:57Z" 2024-06-15T09:29:57.930 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:29:57.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:30:02.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:02.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:02.933 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:02.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:02.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:03.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:03.035+0000 7f439a2eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:03.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:03.039+0000 7f439a2eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.060 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:03.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:03.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:03.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:03.064 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:03Z" 2024-06-15T09:30:03.065 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:03.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:30:08.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:08.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:08.067 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:08.105 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:08.105 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:08.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:08.158+0000 7f401a4a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:08.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:08.158+0000 7f401a4a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:08.181 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:08.181 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:08.181 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:08.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:08.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:08.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:08.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:08.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:08Z" 2024-06-15T09:30:08.187 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:08.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:30:13.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:13.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:13.191 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:13.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:13.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:13.276 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:13.274+0000 7f6b46e04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:13.277 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:13.274+0000 7f6b46e04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.298 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:13.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:13.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:13.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:13Z" 2024-06-15T09:30:13.303 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:13.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:30:18.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:18.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:18.305 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:18.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:18.352 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:18.412 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:18.410+0000 7f7c0ae80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:18.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:18.410+0000 7f7c0ae80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:18.432 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:18.432 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:18.432 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:18.432 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:18.432 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:18.433 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:18.434 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.895427_569.1", 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:18.435 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.895427Z", 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:18.436 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:18Z" 2024-06-15T09:30:18.437 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:18.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.895427_569.1 2024-06-15T09:30:23.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:23.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:23.440 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:23.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:23.484 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:23.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:23.538+0000 7fe420d07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:23.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:23.538+0000 7fe420d07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:23.566 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:23.566 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:23.566 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:23.566 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:23.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:23.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:23.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:23.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:23Z" 2024-06-15T09:30:23.571 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:23.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.955581_573.1 2024-06-15T09:30:28.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:28.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:28.574 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:28.617 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:28.617 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:28.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:28.658+0000 7fb1157dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:28.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:28.662+0000 7fb1157dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:28.681 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:28.681 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:28.681 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:28.681 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:28.681 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:28.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:28.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:28.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:28.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:28Z" 2024-06-15T09:30:28.686 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:28.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:30:33.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:33.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:33.688 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:33.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:33.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:33.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:33.838+0000 7f9fca93ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:33.840 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:33.838+0000 7f9fca93ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:33.864 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:33.864 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:33.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:33.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:30:33.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437220.458611_106.1", 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:40:20.458611Z", 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:33.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:33Z" 2024-06-15T09:30:33.869 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:33.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:30:38.874 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2024-06-15T09:30:38.875 INFO:rgw_multi.tests:create role zone=test-zone1 name=roles-xdvrxv-1 2024-06-15T09:30:38.906 INFO:rgw_multi.tests:create role zone=test-zone2 name=roles-xdvrxv-2 2024-06-15T09:30:38.957 INFO:rgw_multi.tests:create role zone=test-zone3 name=roles-xdvrxv-3 2024-06-15T09:30:39.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:39.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:39.003 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:39.045 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:39.130 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443531.023632_1339.1", 2024-06-15T09:30:39.228 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:25:31.023632Z" 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:30:39.229 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:30:39.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:30:39.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:39.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:39.232 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:39.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:39.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:39.325 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:39.322+0000 7fbd579aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:39.326 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:39.322+0000 7fbd579aea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:39.346 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.887063_620.1", 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.887063Z", 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:39Z" 2024-06-15T09:30:39.352 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:39.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1718443838.930258_623.1 target=1_1718443838.887063_620.1, shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:30:44.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:44.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:44.354 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:44.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:44.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:44.455 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:44.454+0000 7f906ddb7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:44.456 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:44.454+0000 7f906ddb7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:44.480 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:44.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:30:44.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:30:44.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:44.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:44.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:44.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:44.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:44Z" 2024-06-15T09:30:44.487 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:44.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:30:49.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:49.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:49.489 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:49.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:49.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:49.582 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:49.582+0000 7f67258eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:49.583 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:49.586+0000 7f67258eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.624 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:49.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.628 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:49.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:49.629 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:49.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:49.629 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:49Z" 2024-06-15T09:30:49.629 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:49.630 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:30:54.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:54.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:54.631 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:54.674 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:54.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:54.715 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:54.718+0000 7f2450d93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:54.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:54.718+0000 7f2450d93a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:54.734 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:54.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.736 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:54.737 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.738 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:54.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:54.739 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:54.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:54.739 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:54Z" 2024-06-15T09:30:54.739 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:54.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438538.004281_580.1 2024-06-15T09:30:59.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:59.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:30:59.742 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:30:59.800 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:30:59.800 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:30:59.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:59.873+0000 7f5e1b4d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:59.873 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:30:59.873+0000 7f5e1b4d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:30:59.898 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:30:59.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:59.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:30:59.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:30:59Z" 2024-06-15T09:30:59.903 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:30:59.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:04.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:04.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:04.907 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:04.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:04.950 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:04.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:04.993+0000 7f57f2916a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:04.994 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:04.993+0000 7f57f2916a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:05.013 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:05.013 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:05.013 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:05.013 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:05.013 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:05.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:05.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:05.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:05.017 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:04Z" 2024-06-15T09:31:05.018 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:05.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:10.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:10.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:10.020 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:10.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:10.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:10.122 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:10.121+0000 7fee84a0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:10.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:10.125+0000 7fee84a0ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:10.145 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:10.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:10.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:10.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:10.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:10.150 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:10Z" 2024-06-15T09:31:10.151 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:10.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:15.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:15.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:15.154 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:15.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:15.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:15.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:15.253+0000 7f6abb01fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:15.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:15.253+0000 7f6abb01fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:15.275 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:15.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:15.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.279 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:15.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:15.280 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:15.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:15.280 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:15Z" 2024-06-15T09:31:15.280 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:15.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:20.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:20.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:20.283 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:20.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:20.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:20.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:20.389+0000 7fcff6736a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:20.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:20.389+0000 7fcff6736a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:20.412 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:20.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:20.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:20.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:20.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:20Z" 2024-06-15T09:31:20.417 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:20.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:25.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:25.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:25.420 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:25.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:25.463 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:25.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:25.517+0000 7efc88a8da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:25.518 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:25.521+0000 7efc88a8da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:25.543 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:25.549 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:25Z" 2024-06-15T09:31:25.549 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:25.550 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:30.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:30.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:30.552 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:30.596 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:30.596 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:30.644 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:30.645+0000 7f310ef12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:30.645 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:30.645+0000 7f310ef12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:30.667 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:30.672 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:30Z" 2024-06-15T09:31:30.673 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:30.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:31:35.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:35.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:35.675 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:35.719 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:35.719 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:35.779 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:35.781+0000 7fb0dd6d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:35.780 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:35.781+0000 7fb0dd6d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:35.804 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:35.804 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:35.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:35.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:35.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:35.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:35.809 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:35Z" 2024-06-15T09:31:35.810 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:35.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:31:40.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:40.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:40.812 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:40.855 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:40.856 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:40.898 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:40.897+0000 7f66e434aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:40.898 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:40.901+0000 7f66e434aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:40.919 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:40.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:40.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:40.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:40.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:40Z" 2024-06-15T09:31:40.924 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:40.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:45.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:45.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:45.927 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:45.970 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:45.970 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:46.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:46.013+0000 7f1a7019ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:46.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:46.017+0000 7f1a7019ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:46.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:46.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:46.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:46.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:46Z" 2024-06-15T09:31:46.039 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:46.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:51.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:51.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:51.042 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:51.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:51.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:51.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:51.132+0000 7fe103b04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:51.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:51.132+0000 7fe103b04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:51.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:51.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:51.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:51.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:51.155 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:51Z" 2024-06-15T09:31:51.156 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:51.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:31:56.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:56.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:31:56.159 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:31:56.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:31:56.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:31:56.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:56.256+0000 7f8205d1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:56.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:31:56.256+0000 7f8205d1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:56.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:56.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:56.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:56.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:31:56.283 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:31:56.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:31:56Z" 2024-06-15T09:31:56.285 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:31:56.287 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:01.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:01.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:01.288 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:01.332 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:01.332 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:01.381 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:01.380+0000 7fd2b9d12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:01.382 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:01.380+0000 7fd2b9d12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:01.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:01.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.123499_586.1", 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:01.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.123499Z", 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:01.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:01Z" 2024-06-15T09:32:01.412 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:01.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.123499_586.1 2024-06-15T09:32:06.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:06.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:06.415 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:06.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:06.459 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:06.508 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:06.508+0000 7fc37eca0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:06.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:06.508+0000 7fc37eca0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:06.536 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:06.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:06.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:06.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:06.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:06Z" 2024-06-15T09:32:06.541 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:06.542 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:11.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:11.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:11.543 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:11.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:11.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:11.645 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:11.644+0000 7efc1edb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:11.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:11.648+0000 7efc1edb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:11.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:11.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.809649_674.1", 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.809649Z", 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.678 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:11.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:11.679 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:11.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:11.679 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:11Z" 2024-06-15T09:32:11.679 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:11.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438845.809649_674.1 2024-06-15T09:32:16.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:16.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:16.681 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:16.724 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:16.725 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:16.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:16.768+0000 7ff889e13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:16.771 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:16.772+0000 7ff889e13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:16.793 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:16.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:16.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.796 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:16.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:16Z" 2024-06-15T09:32:16.798 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:16.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.076452_581.1 2024-06-15T09:32:21.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:21.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:21.800 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:21.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:21.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:21.897 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:21.896+0000 7fe071e77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:21.898 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:21.896+0000 7fe071e77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:21.918 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:21.918 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:21.918 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:21.918 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:21.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:21.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:32:21.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:21.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:21Z" 2024-06-15T09:32:21.923 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:21.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.076452_581.1 2024-06-15T09:32:26.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:26.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:26.925 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:26.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:26.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:27.026 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:27.024+0000 7f8445907a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:27.026 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:27.024+0000 7f8445907a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.046 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:27.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:27.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:27.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:27.050 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:27Z" 2024-06-15T09:32:27.051 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:27.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:32.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:32.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:32.053 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:32.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:32.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:32.159 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:32.160+0000 7feb05992a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:32.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:32.160+0000 7feb05992a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:32.184 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:32.184 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:32.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:32.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:32.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:32.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:32Z" 2024-06-15T09:32:32.189 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:32.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:37.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:37.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:37.192 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:37.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:37.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:37.294 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:37.292+0000 7fb97a5baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:37.294 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:37.292+0000 7fb97a5baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:37.318 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:37.318 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:37.318 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:37.318 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:37.318 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:37.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:37.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:37.321 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:37.322 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:37Z" 2024-06-15T09:32:37.323 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:37.324 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:42.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:42.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:42.326 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:42.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:42.370 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:42.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:42.419+0000 7ff457987a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:42.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:42.419+0000 7ff457987a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:42.445 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:42.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:42.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.449 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:42.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:42.450 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:42.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:42.450 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:42Z" 2024-06-15T09:32:42.450 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:42.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:47.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:47.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:47.452 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:47.496 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:47.496 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:47.544 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:47.543+0000 7f0ec9b6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:47.545 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:47.543+0000 7f0ec9b6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:47.564 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:47.564 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:47.564 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:47.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:47.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.116859_585.1", 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.116859Z", 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:47.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:47.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:47Z" 2024-06-15T09:32:47.569 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:47.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.116859_585.1 2024-06-15T09:32:52.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:52.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:52.571 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:52.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:52.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:52.673 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:52.671+0000 7f83edc09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:52.674 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:52.671+0000 7f83edc09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:52.699 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:52.699 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:52.699 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:52.699 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:52.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:52.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:52.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:52.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:52.704 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:52Z" 2024-06-15T09:32:52.705 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:52.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:32:57.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:57.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:32:57.707 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:32:57.752 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:32:57.753 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:32:57.796 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:57.795+0000 7f972995ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:57.797 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:32:57.795+0000 7f972995ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.818 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:57.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:57.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439520.008384_809.1", 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:18:40.008384Z", 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:32:57.821 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:32:57.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:32:57Z" 2024-06-15T09:32:57.823 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:32:57.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718439520.008384_809.1 2024-06-15T09:33:02.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:02.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:02.825 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:02.869 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:02.870 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:02.919 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:02.919+0000 7fe7bbe66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:02.920 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:02.919+0000 7fe7bbe66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:02.939 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:02.939 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:02.939 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:02.939 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:02.939 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:02.940 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:02.941 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:33:02.942 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:02.943 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:02Z" 2024-06-15T09:33:02.944 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:02.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.076452_581.1 2024-06-15T09:33:07.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:07.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:07.947 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:07.990 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:07.991 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:08.032 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:08.031+0000 7f4fae313a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:08.033 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:08.031+0000 7f4fae313a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:08.051 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:08.051 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:08.051 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:08.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:08.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T09:33:08.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:08.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:08Z" 2024-06-15T09:33:08.056 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:08.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438538.004281_580.1 2024-06-15T09:33:13.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:13.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:13.058 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:13.103 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:13.103 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:13.147 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:13.147+0000 7f2d3a359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:13.148 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:13.147+0000 7f2d3a359a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:13.166 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:13.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:13.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:13.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:13.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:13Z" 2024-06-15T09:33:13.171 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:13.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:18.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:18.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:18.173 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:18.216 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:18.216 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:18.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:18.255+0000 7f5a5e4e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:18.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:18.259+0000 7f5a5e4e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:18.283 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:18.283 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:18.283 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:18.284 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:18.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:18.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:18.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:18Z" 2024-06-15T09:33:18.288 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:18.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.152646_587.1 2024-06-15T09:33:23.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:23.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:23.290 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:23.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:23.344 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:23.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:23.399+0000 7f9c50c67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:23.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:23.399+0000 7f9c50c67a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:23.427 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:23.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:23.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:23.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:23.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:23Z" 2024-06-15T09:33:23.432 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:23.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.152646_587.1 2024-06-15T09:33:28.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:28.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:28.434 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:28.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:28.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:28.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:28.519+0000 7f378a3b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:28.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:28.523+0000 7f378a3b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:28.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:28.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:28.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:28.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:28.550 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:28.552 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:28.552 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:28.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:28.552 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:28Z" 2024-06-15T09:33:28.552 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:28.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:33.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:33.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:33.554 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:33.598 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:33.598 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:33.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:33.643+0000 7f32daacfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:33.643 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:33.643+0000 7f32daacfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:33.664 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:33.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:33.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:33.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:33.668 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:33Z" 2024-06-15T09:33:33.669 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:33.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:38.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:38.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:38.672 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:38.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:38.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:38.758 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:38.758+0000 7f67e5c76a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:38.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:38.758+0000 7f67e5c76a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:38.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:38.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:38.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:38.783 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:38Z" 2024-06-15T09:33:38.784 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:38.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:43.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:43.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:43.787 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:43.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:43.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:43.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:43.886+0000 7f45149d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:43.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:43.890+0000 7f45149d3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.917 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:43.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:43.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:43.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:43Z" 2024-06-15T09:33:43.922 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:43.924 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:48.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:48.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:48.926 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:48.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:48.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:49.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:49.018+0000 7fabf22efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:49.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:49.018+0000 7fabf22efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:49.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:49.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:49.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:49.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:49.046 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:49.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:49.047 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:49Z" 2024-06-15T09:33:49.047 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:49.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:33:54.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:54.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:54.049 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:54.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:54.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:54.151 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:54.150+0000 7ff04cf3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:54.152 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:54.150+0000 7ff04cf3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:54.179 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:54.179 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:54.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:54.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:54.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:54.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:54Z" 2024-06-15T09:33:54.184 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:54.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:33:59.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:59.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:33:59.187 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:33:59.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:33:59.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:33:59.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:59.282+0000 7fd5674a9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:59.286 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:33:59.282+0000 7fd5674a9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:33:59.306 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:59.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:33:59.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:59.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:33:59.310 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:33:59Z" 2024-06-15T09:33:59.311 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:33:59.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:04.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:04.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:04.314 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:04.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:04.357 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:04.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:04.398+0000 7f50bb709a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:04.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:04.398+0000 7f50bb709a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:04.422 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:04.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:04.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:04.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:04.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:04Z" 2024-06-15T09:34:04.427 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:04.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:09.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:09.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:09.429 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:09.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:09.471 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:09.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:09.514+0000 7fd1950fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:09.518 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:09.514+0000 7fd1950fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:09.538 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:09.538 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:09.538 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:09.538 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:09.538 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:09.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:09.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:09.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:09Z" 2024-06-15T09:34:09.543 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:09.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:14.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:14.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:14.546 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:14.589 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:14.589 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:14.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:14.638+0000 7ff3157e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:14.643 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:14.642+0000 7ff3157e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:14.661 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:14.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:14.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:14.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:14.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:14Z" 2024-06-15T09:34:14.666 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:14.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:34:19.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:19.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:19.669 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:19.719 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:19.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:19.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:19.770+0000 7fedddceca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:19.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:19.770+0000 7fedddceca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:19.788 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:19.788 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:19.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:19.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:19.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:19.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.793 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:19.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:19.794 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:19.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:19.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:19Z" 2024-06-15T09:34:19.794 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:19.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:24.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:24.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:24.797 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:24.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:24.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:24.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:24.906+0000 7fd1887e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:24.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:24.910+0000 7fd1887e5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.933 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:24.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:24.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:24.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:24.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:24.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:24.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:24.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:24.938 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:24.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:24.939 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:24.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.939 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:24.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:24.940 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:24.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:24.940 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:24Z" 2024-06-15T09:34:24.940 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:24.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:29.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:29.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:29.942 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:29.987 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:29.988 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:30.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:30.029+0000 7ff69ce47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:30.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:30.033+0000 7ff69ce47a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:30.051 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:30.051 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:30.051 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:30.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:30.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:30.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:30.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:30Z" 2024-06-15T09:34:30.056 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:30.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:35.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:35.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:35.059 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:35.102 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:35.102 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:35.154 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:35.153+0000 7f99012e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:35.155 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:35.157+0000 7f99012e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:35.178 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:35.179 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:35.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:35.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:35.182 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:35Z" 2024-06-15T09:34:35.183 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:35.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:40.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:40.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:40.185 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:40.227 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:40.227 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:40.286 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:40.285+0000 7f3f3efbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:40.287 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:40.289+0000 7f3f3efbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:40.312 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:40.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:40.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:40.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:40.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:40Z" 2024-06-15T09:34:40.317 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:40.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:45.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:45.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:45.320 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:45.365 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:45.365 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:45.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:45.413+0000 7f278f751a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:45.414 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:45.413+0000 7f278f751a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.440 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:45.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:45.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:45.444 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:45.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:45.445 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:45Z" 2024-06-15T09:34:45.445 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:45.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.076452_581.1 2024-06-15T09:34:50.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:50.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:50.448 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:50.492 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:50.492 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:50.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:50.553+0000 7f789f3d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:50.555 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:50.557+0000 7f789f3d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:50.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:50.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:50.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:50.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:50.583 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:50Z" 2024-06-15T09:34:50.583 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:50.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:34:55.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:55.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:34:55.585 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:34:55.630 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:34:55.630 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:34:55.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:55.681+0000 7fae71dbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:55.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:34:55.681+0000 7fae71dbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:34:55.705 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:34:55.705 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:34:55.706 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:55.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:55.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:34:55.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:34:55Z" 2024-06-15T09:34:55.710 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:34:55.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:35:00.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:00.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:00.714 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:00.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:00.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:00.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:00.805+0000 7f4532053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:00.806 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:00.805+0000 7f4532053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:00.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:00.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:00.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:00.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.835 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:00Z" 2024-06-15T09:35:00.836 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:00.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:05.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:05.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:05.838 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:05.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:05.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:05.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:05.925+0000 7feb57165a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:05.926 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:05.925+0000 7feb57165a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:05.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:05.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:05.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:05Z" 2024-06-15T09:35:05.957 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:05.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:10.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:10.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:10.959 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:11.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:11.001 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:11.059 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:11.061+0000 7f518737aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:11.060 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:11.061+0000 7f518737aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:11.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:11.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:11.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:11.086 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:11.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:11.087 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:11Z" 2024-06-15T09:35:11.087 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:11.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:16.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:16.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:16.089 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:16.133 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:16.133 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:16.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:16.200+0000 7fa5ed514a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:16.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:16.200+0000 7fa5ed514a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:16.220 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:16.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:16.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:16.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:16.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:16.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:16Z" 2024-06-15T09:35:16.225 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:16.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:21.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:21.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:21.371 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:21.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:21.415 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:21.474 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:21.472+0000 7fa2a7e82a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:21.475 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:21.476+0000 7fa2a7e82a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:21.505 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:35:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:21Z" 2024-06-15T09:35:21.511 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:21.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:35:26.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:26.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:26.513 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:26.553 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:26.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:26.620+0000 7f440f27aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:26.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:26.620+0000 7f440f27aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:26.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:26.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:26.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:26.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:26.649 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:26.650 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:26Z" 2024-06-15T09:35:26.650 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:26.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:31.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:31.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:31.652 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:31.696 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:31.696 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:31.750 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:31.748+0000 7fb030c5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:31.750 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:31.748+0000 7fb030c5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:31.775 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:31.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:31.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:31.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:31Z" 2024-06-15T09:35:31.780 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:31.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:36.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:36.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:36.783 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:36.827 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:36.828 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:36.869 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:36.868+0000 7fa8cac00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:36.870 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:36.868+0000 7fa8cac00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:36.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:36.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:36.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.116859_585.1", 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.116859Z", 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:36.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:36.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:36Z" 2024-06-15T09:35:36.896 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:36.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438539.116859_585.1 2024-06-15T09:35:41.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:41.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:41.898 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:41.942 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:41.942 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:41.997 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:41.996+0000 7fb48c60ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:41.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:41.996+0000 7fb48c60ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:42.024 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:42.025 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:42.026 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:42.027 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:42.028 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:42Z" 2024-06-15T09:35:42.029 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:42.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718443531.023632_1339.1 target=1_1718438536.979596_577.1 2024-06-15T09:35:47.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... FAIL 2024-06-15T09:35:47.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:47.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:47.080 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:47.139 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:47.139 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:47.304 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:35:47.334 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718444147.076158_1399.1", 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:35:47.076158Z" 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:35:47.335 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:35:47.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:35:47.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:47.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:47.337 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:47.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:47.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:47.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:47.428+0000 7ff26355da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:47.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:47.428+0000 7ff26355da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:47.446 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:47.446 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:47.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:47.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:35:47.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:47.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:47Z" 2024-06-15T09:35:47.451 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:47.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:35:52.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:52.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:52.454 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:52.499 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:52.499 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:52.559 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:52.560+0000 7fbf6dc54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:52.560 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:52.560+0000 7fbf6dc54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:52.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:52.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:52.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:52.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:52Z" 2024-06-15T09:35:52.591 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:52.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:35:57.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:57.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:35:57.594 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:35:57.637 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:35:57.637 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:35:57.688 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:57.688+0000 7f722c3fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:57.689 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:35:57.688+0000 7f722c3fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:57.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:57.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:35:57.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:35:57.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:35:57.718 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:35:57.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:35:57.719 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:35:57Z" 2024-06-15T09:35:57.719 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:35:57.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.179989_590.1 2024-06-15T09:36:02.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:02.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:02.721 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:02.773 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:02.773 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:02.845 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:02.844+0000 7fdd90705a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:02.846 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:02.848+0000 7fdd90705a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:02.870 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:02.870 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:02.871 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:02.872 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:02.873 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.874 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:02.875 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:02Z" 2024-06-15T09:36:02.876 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:02.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:07.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:07.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:07.879 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:07.922 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:07.922 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:07.963 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:07.963+0000 7f9445589a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:07.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:07.963+0000 7f9445589a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:07.982 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:07.982 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:07.982 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:07.983 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:07.984 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:07.985 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:07.986 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:07Z" 2024-06-15T09:36:07.987 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:07.988 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:12.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:12.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:12.990 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:13.040 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:13.040 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:13.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:13.095+0000 7feb1d507a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:13.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:13.095+0000 7feb1d507a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:13.116 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:13.116 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:13.116 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:13.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:13.118 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:13.119 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:13.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:13Z" 2024-06-15T09:36:13.121 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:13.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:18.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:18.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:18.123 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:18.166 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:18.166 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:18.214 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:18.211+0000 7f1b9386ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:18.214 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:18.215+0000 7f1b9386ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:18.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:18.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:18.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.239 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:18.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:18.240 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:18.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:18.240 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:18Z" 2024-06-15T09:36:18.240 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:18.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:23.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:23.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:23.242 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:23.285 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:23.285 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:23.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:23.327+0000 7fe0613e0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:23.329 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:23.327+0000 7fe0613e0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:23.347 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:23.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:23.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:23.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:23.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:23Z" 2024-06-15T09:36:23.352 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:23.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:28.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:28.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:28.354 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:28.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:28.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:28.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:28.443+0000 7f1a1b561a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:28.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:28.443+0000 7f1a1b561a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:28.467 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:28.467 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:28.468 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:28.469 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:28.470 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.083531_582.1", 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.083531Z", 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:28.471 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:28.472 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:28.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:28.473 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:28Z" 2024-06-15T09:36:28.473 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:28.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.083531_582.1 2024-06-15T09:36:33.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:33.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:33.476 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:33.519 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:33.519 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:33.580 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:33.579+0000 7f40349f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:33.581 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:33.579+0000 7f40349f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:33.606 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:33.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:33.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:33.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.610 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.611 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:33.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:33.612 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:33.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:33.612 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:33Z" 2024-06-15T09:36:33.612 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:33.613 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:38.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:38.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:38.614 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:38.657 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:38.658 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:38.720 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:38.719+0000 7fae32ae6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:38.721 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:38.719+0000 7fae32ae6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:38.741 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:38.745 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:38Z" 2024-06-15T09:36:38.746 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:38.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:43.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:43.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:43.749 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:43.787 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:43.788 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:43.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:43.827+0000 7f64f56c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:43.831 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:43.831+0000 7f64f56c3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:43.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:43.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:43.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:43.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:43Z" 2024-06-15T09:36:43.853 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:43.854 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:48.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:48.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:48.856 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:48.900 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:48.900 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:48.943 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:48.943+0000 7faa9278aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:48.944 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:48.943+0000 7faa9278aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:48.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:48.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:48.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:48.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:48.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:48Z" 2024-06-15T09:36:48.968 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:48.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:53.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:53.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:53.971 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:54.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:54.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:54.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:54.067+0000 7f6b0410da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:54.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:54.071+0000 7f6b0410da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:54.095 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:54.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:54.095 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:54.095 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:54.095 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:54.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:54.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:54.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:54.099 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:54Z" 2024-06-15T09:36:54.100 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:54.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:36:59.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:59.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:36:59.102 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:36:59.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:36:59.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:36:59.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:59.198+0000 7f570fcf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:59.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:36:59.198+0000 7f570fcf8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:36:59.220 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:36:59.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:59.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:59.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:59.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:36:59.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:36:59Z" 2024-06-15T09:36:59.225 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:36:59.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:37:04.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:04.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:04.228 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:04.272 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:04.272 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:04.314 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:04.310+0000 7f60d3423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:04.315 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:04.314+0000 7f60d3423a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:04.335 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:04.335 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:04.335 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:04.336 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:04.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.955581_573.1", 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.955581Z", 2024-06-15T09:37:04.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:04.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:04Z" 2024-06-15T09:37:04.340 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:04.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.955581_573.1 2024-06-15T09:37:09.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:09.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:09.342 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:09.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:09.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:09.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:09.430+0000 7f6abb8cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:09.432 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:09.430+0000 7f6abb8cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:09.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:09.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.123499_586.1", 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.123499Z", 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:09.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:09.456 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:09.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:09.456 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:09Z" 2024-06-15T09:37:09.456 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:09.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.123499_586.1 2024-06-15T09:37:14.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:14.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:14.458 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:14.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:14.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:14.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:14.542+0000 7f506d04ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:14.544 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:14.542+0000 7f506d04ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:14.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:14.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:14.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:14Z" 2024-06-15T09:37:14.566 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:14.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:37:19.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:19.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:19.569 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:19.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:19.613 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:19.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:19.662+0000 7f2c70a77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:19.667 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:19.666+0000 7f2c70a77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:19.691 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:19.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:19.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:19.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:19.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.696 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:19.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:19.697 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:19.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:19.697 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:19Z" 2024-06-15T09:37:19.697 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:19.698 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:37:24.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:24.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:24.700 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:24.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:24.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:24.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:24.782+0000 7f6cdef4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:24.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:24.782+0000 7f6cdef4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.805 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:24.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:24.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:24.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:24.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:24.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:24.811 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:24.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:24.811 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:24Z" 2024-06-15T09:37:24.811 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:24.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:37:29.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:29.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:29.813 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:29.856 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:29.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:29.900 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:29.898+0000 7f7ab7356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:29.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:29.898+0000 7f7ab7356a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:29.920 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:29.920 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:29.920 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:29.920 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:29.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:29.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:29.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:29.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:29Z" 2024-06-15T09:37:29.925 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:29.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:37:34.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:34.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:34.928 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:34.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:34.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:35.035 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:35.034+0000 7efc23168a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:35.036 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:35.034+0000 7efc23168a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:35.069 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:35.069 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:35.069 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:35.069 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:35.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:35.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:35.074 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:35.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:35.074 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:35Z" 2024-06-15T09:37:35.074 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:35.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:37:40.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:40.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:40.076 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:40.118 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:40.119 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:40.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:40.182+0000 7f9a1dd4ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:40.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:40.182+0000 7f9a1dd4ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:40.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:40.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:40.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:40.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:40.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:40.205 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:40Z" 2024-06-15T09:37:40.205 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:40.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438538.004281_580.1 2024-06-15T09:37:45.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:45.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:45.207 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:45.251 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:45.252 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:45.310 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:45.310+0000 7fb98dd04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:45.311 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:45.314+0000 7fb98dd04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:45.388 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:45.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:45.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:45.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:45.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:45.393 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:45Z" 2024-06-15T09:37:45.394 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:45.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:37:50.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:50.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:50.396 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:50.440 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:50.440 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:50.485 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:50.485+0000 7f85b1790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:50.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:50.485+0000 7f85b1790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:50.504 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:50.504 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:50.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:50.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.114025_584.1", 2024-06-15T09:37:50.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.114025Z", 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:50.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:50.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:50.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:50Z" 2024-06-15T09:37:50.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:50.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.114025_584.1 2024-06-15T09:37:55.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:55.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:37:55.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:37:55.564 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:37:55.564 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:37:55.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:55.617+0000 7f69afabaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:55.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:37:55.617+0000 7f69afabaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:55.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:55.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:37:55.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:37:55.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:37:55.643 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:37:55.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:37:55.643 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:37:55Z" 2024-06-15T09:37:55.643 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:37:55.644 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:00.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:00.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:00.646 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:00.688 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:00.689 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:00.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:00.733+0000 7f43fb0d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:00.735 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:00.737+0000 7f43fb0d1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:00.758 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:00.759 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:00.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:00.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:00.762 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:00Z" 2024-06-15T09:38:00.763 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:00.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:05.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:05.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:05.765 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:05.808 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:05.808 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:05.855 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:05.857+0000 7fe7b3c78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:05.856 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:05.857+0000 7fe7b3c78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:05.876 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:05.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:05.878 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:05.879 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:05.880 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:05.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:05.882 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:05.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:05.882 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:05Z" 2024-06-15T09:38:05.882 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:05.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:10.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:10.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:10.884 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:10.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:10.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:10.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:10.993+0000 7efc0ef3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:10.994 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:10.993+0000 7efc0ef3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:11.018 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:11.018 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:11.018 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:11.018 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:11.018 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:10Z" 2024-06-15T09:38:11.023 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:11.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:16.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:16.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:16.026 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:16.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:16.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:16.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:16.129+0000 7f38e04b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:16.131 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:16.133+0000 7f38e04b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:16.151 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:16.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:16.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:16.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:16.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:16Z" 2024-06-15T09:38:16.156 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:16.157 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:21.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:21.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:21.158 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:21.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:21.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:21.244 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:21.245+0000 7f5ca1222a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:21.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:21.245+0000 7f5ca1222a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:21.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:21.265 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:21.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:21.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439520.005312_808.1", 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:18:40.005312Z", 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:21.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:21.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:21.270 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:21Z" 2024-06-15T09:38:21.271 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:21.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718439520.005312_808.1 2024-06-15T09:38:26.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:26.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:26.273 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:26.318 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:26.318 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:26.378 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:26.377+0000 7fc805efca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:26.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:26.381+0000 7fc805efca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:26.405 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:26.405 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:26.405 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:26.406 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:26.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439520.005312_808.1", 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:26.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:18:40.005312Z", 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:26.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:26.410 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:26.413 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:26.414 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:26Z" 2024-06-15T09:38:26.414 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:26.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718439520.005312_808.1 2024-06-15T09:38:31.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:31.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:31.417 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:31.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:31.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:31.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:31.517+0000 7fa20f37da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:31.518 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:31.517+0000 7fa20f37da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:31.537 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:31.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:31.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:31.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:31.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:31.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:31.543 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:31Z" 2024-06-15T09:38:31.543 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:31.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:36.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:36.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:36.545 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:36.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:36.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:36.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:36.633+0000 7f1be0294a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:36.632 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:36.633+0000 7f1be0294a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:36.658 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:36.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:36.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:36.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:36.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:36Z" 2024-06-15T09:38:36.663 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:36.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:41.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:41.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:41.666 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:41.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:41.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:41.761 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:41.760+0000 7f6d4318ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:41.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:41.760+0000 7f6d4318ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:41.782 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:41.782 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:41.782 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:41.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:41.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:41.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:41.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:41Z" 2024-06-15T09:38:41.787 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:41.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:38:46.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:46.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:46.790 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:46.833 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:46.834 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:46.876 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:46.876+0000 7fc83d4bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:46.877 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:46.876+0000 7fc83d4bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:46.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:46.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:46.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:46.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:46.902 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:46Z" 2024-06-15T09:38:46.902 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:46.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:38:51.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:51.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:51.904 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:51.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:51.947 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:51.995 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:51.996+0000 7f8a4a70da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:51.996 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:51.996+0000 7f8a4a70da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:52.018 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:52.019 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:52.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:52.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:52.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:52.023 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:52Z" 2024-06-15T09:38:52.023 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:52.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:38:57.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:57.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:38:57.025 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:38:57.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:38:57.068 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:38:57.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:57.128+0000 7fad21805a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:57.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:38:57.128+0000 7fad21805a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:38:57.150 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:38:57.150 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:38:57.150 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:38:57.150 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:38:57.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:57.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:57.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:57.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:38:57Z" 2024-06-15T09:38:57.155 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:38:57.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:39:02.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:02.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:02.158 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:02.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:02.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:02.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:02.244+0000 7ff4e8559a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:02.246 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:02.244+0000 7ff4e8559a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:02.264 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:02.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:02.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:02.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:02.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:02Z" 2024-06-15T09:39:02.269 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:02.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:07.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:07.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:07.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:07.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:07.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:07.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:07.372+0000 7f99c34eba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:07.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:07.372+0000 7f99c34eba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:07.391 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:07.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:07.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.760139_669.1", 2024-06-15T09:39:07.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.760139Z", 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:07.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:07Z" 2024-06-15T09:39:07.396 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:07.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438845.760139_669.1 2024-06-15T09:39:12.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:12.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:12.398 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:12.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:12.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:12.508 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:12.508+0000 7fac23c14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:12.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:12.508+0000 7fac23c14a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:12.526 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:12.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:12.526 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:12.526 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:12.526 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:12.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:12.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:12.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:12Z" 2024-06-15T09:39:12.531 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:12.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:17.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:17.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:17.533 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:17.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:17.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:17.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:17.620+0000 7f5410293a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:17.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:17.620+0000 7f5410293a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:17.639 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:17.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:17.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.643 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:17.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:17.644 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:17.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:17.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:17Z" 2024-06-15T09:39:17.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:17.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:22.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:22.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:22.646 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:22.689 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:22.689 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:22.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:22.736+0000 7f0f3f088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:22.735 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:22.736+0000 7f0f3f088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:22.754 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:22.754 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:22.754 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:22.754 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:22.754 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:22.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:22.757 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:22.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:22Z" 2024-06-15T09:39:22.759 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:22.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:39:27.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:27.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:27.762 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:27.805 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:27.805 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:27.868 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:27.868+0000 7fed843f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:27.869 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:27.868+0000 7fed843f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:27.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:27.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:27.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:27.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:27.901 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:27Z" 2024-06-15T09:39:27.902 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:27.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:32.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:32.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:32.904 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:32.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:32.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:33.006 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:33.003+0000 7fc6df715a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:33.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:33.007+0000 7fc6df715a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:33.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:33.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:33.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:33.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:33.036 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:33.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:33.037 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:33Z" 2024-06-15T09:39:33.037 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:33.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:38.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:38.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:38.039 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:38.090 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:38.090 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:38.140 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:38.139+0000 7ff93d35ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:38.141 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:38.139+0000 7ff93d35ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:38.164 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:38.164 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:38.164 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:38.164 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:38.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:38.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:38.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:38.168 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:38Z" 2024-06-15T09:39:38.169 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:38.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:43.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:43.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:43.171 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:43.215 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:43.216 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:43.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:43.255+0000 7f78c6049a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:43.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:43.255+0000 7f78c6049a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:43.276 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:43.276 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:43.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:43.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:43.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:43.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:43Z" 2024-06-15T09:39:43.281 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:43.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:48.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:48.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:48.283 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:48.331 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:48.331 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:48.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:48.379+0000 7f14552d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:48.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:48.379+0000 7f14552d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:48.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:48.399 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:48.400 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:48.401 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:48Z" 2024-06-15T09:39:48.402 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:48.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:53.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:53.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:53.405 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:53.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:53.478 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:53.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:53.539+0000 7f80feee1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:53.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:53.539+0000 7f80feee1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:53.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:53.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:53.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:53.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:53.570 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:53.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:53.570 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:53Z" 2024-06-15T09:39:53.570 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:53.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:39:58.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:58.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:39:58.573 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:39:58.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:39:58.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:39:58.660 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:58.659+0000 7fdd9ec4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:58.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:39:58.659+0000 7fdd9ec4da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:39:58.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:39:58.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:58.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:39:58.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:39:58Z" 2024-06-15T09:39:58.691 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:39:58.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:03.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:03.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:03.693 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:03.777 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:03.777 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:03.831 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:03.831+0000 7fb4e49fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:03.832 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:03.831+0000 7fb4e49fba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:03.857 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:03.858 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:03.859 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:03.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:03.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.862 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:03.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:03.863 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:03.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:03.863 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:03Z" 2024-06-15T09:40:03.863 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:03.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:08.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:08.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:08.866 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:09.024 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:09.024 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:09.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:09.071+0000 7f208ee95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:09.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:09.071+0000 7f208ee95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:09.090 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:09.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:09.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:09.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:09.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:09Z" 2024-06-15T09:40:09.095 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:09.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.076452_581.1 2024-06-15T09:40:14.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:14.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:14.098 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:14.140 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:14.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:14.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:14.227+0000 7f4d10e96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:14.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:14.227+0000 7f4d10e96a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.256 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:14.257 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:14.258 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:14.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:14.260 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:14.261 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:14Z" 2024-06-15T09:40:14.261 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:14.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:19.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:19.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:19.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:19.477 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:19.477 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:19.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:19.523+0000 7fa2123e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:19.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:19.523+0000 7fa2123e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:19.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:19.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:19.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:19Z" 2024-06-15T09:40:19.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:19.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:24.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:24.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:24.549 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:24.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:24.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:24.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:24.654+0000 7f2330efaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:24.657 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:24.654+0000 7f2330efaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:24.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:24.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:24.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.116859_585.1", 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.116859Z", 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:24.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:24.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:24.687 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:24.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:24.687 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:24Z" 2024-06-15T09:40:24.687 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:24.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438539.116859_585.1 2024-06-15T09:40:29.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:29.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:29.690 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:29.961 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:29.962 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:30.012 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:30.010+0000 7f7b24afaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:30.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:30.010+0000 7f7b24afaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:30.038 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:30.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:30.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:30.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:30.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:30.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:30.044 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:30Z" 2024-06-15T09:40:30.044 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:30.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:40:35.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:35.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:35.046 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:35.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:35.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:35.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:35.182+0000 7f6f2a351a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:35.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:35.186+0000 7f6f2a351a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:35.210 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:35.210 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:35.211 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:35.212 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:35.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:35.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:35Z" 2024-06-15T09:40:35.215 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:35.216 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438536.979596_577.1 2024-06-15T09:40:40.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:40.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:40.217 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:40.261 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:40.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:40.318+0000 7fa65b03ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:40.318+0000 7fa65b03ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:40.344 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:40.345 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:40.346 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:40.347 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:40Z" 2024-06-15T09:40:40.348 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:40.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:45.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:45.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:45.349 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:45.392 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:45.393 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:45.434 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:45.434+0000 7f0fbdc31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:45.435 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:45.434+0000 7f0fbdc31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:45.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:45.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:45.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:45.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:45.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:45Z" 2024-06-15T09:40:45.466 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:45.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:50.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:50.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:50.469 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:50.597 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:50.597 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:50.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:50.638+0000 7efc27c88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:50.643 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:50.642+0000 7efc27c88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:50.667 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:50.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:50.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:50.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:50.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:50Z" 2024-06-15T09:40:50.672 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:50.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444147.076158_1399.1 target=1_1718438537.989943_579.1 2024-06-15T09:40:55.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2024-06-15T09:40:55.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:55.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:55.731 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:55.774 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:55.830 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:40:55.852 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:40:55.852 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:40:55.852 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:55.852 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718444455.726175_1433.1", 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:40:55.726175Z" 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:40:55.853 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:40:55.854 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:40:55.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:40:55.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:55.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:40:55.855 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:40:55.924 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:40:55.925 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:40:55.989 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:55.986+0000 7f82b3570a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:55.990 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:40:55.990+0000 7f82b3570a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:40:56.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:56.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:56.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.107383_583.1", 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.107383Z", 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:40:56.015 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:40:55Z" 2024-06-15T09:40:56.016 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:40:56.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.107383_583.1 2024-06-15T09:41:01.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:01.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:01.018 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:01.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:01.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:01.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:01.190+0000 7f171ca3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:01.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:01.190+0000 7f171ca3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.212 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:01.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:01.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:01.216 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:01.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:01.217 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:01Z" 2024-06-15T09:41:01.217 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:01.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.076452_581.1 2024-06-15T09:41:06.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:06.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:06.219 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:06.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:06.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:06.305 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:06.306+0000 7f75bc2fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:06.306 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:06.306+0000 7f75bc2fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:06.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:06.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:06.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:06.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.331 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:06.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:06.332 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:06.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:06.332 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:06Z" 2024-06-15T09:41:06.332 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:06.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:11.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:11.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:11.338 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:11.396 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:11.397 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:11.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:11.446+0000 7f920f5afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:11.445 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:11.446+0000 7f920f5afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:11.463 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:11.463 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:11.463 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:11.463 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:11.463 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:11.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:11.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:11.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:11Z" 2024-06-15T09:41:11.468 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:11.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:16.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:16.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:16.470 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:16.515 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:16.515 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:16.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:16.593+0000 7f10caee3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:16.595 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:16.597+0000 7f10caee3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:16.828 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:16.828 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:16.829 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:16.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:16.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:16.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:16Z" 2024-06-15T09:41:16.833 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:16.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:21.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:21.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:21.836 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:21.963 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:21.963 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:22.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:22.021+0000 7f8f88ec0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:22.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:22.021+0000 7f8f88ec0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:22.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:22.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:22.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.050 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:22.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:22.052 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:22Z" 2024-06-15T09:41:22.053 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:22.054 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.163871_589.1 2024-06-15T09:41:27.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:27.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:27.056 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:27.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:27.100 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:27.141 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:27.141+0000 7fa8b406ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:27.141 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:27.141+0000 7fa8b406ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:27.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:27.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:27.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:27.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:27Z" 2024-06-15T09:41:27.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:27.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.163871_589.1 2024-06-15T09:41:32.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:32.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:32.168 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:32.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:32.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:32.265 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:32.265+0000 7f1a01a24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:32.266 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:32.265+0000 7f1a01a24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:32.286 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:32.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:32.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:32.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:32.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:32Z" 2024-06-15T09:41:32.291 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:32.292 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.261071_591.1 2024-06-15T09:41:37.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:37.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:37.293 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:37.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:37.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:37.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:37.401+0000 7f0c12932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:37.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:37.405+0000 7f0c12932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:37.424 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:37.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:37.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:37.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:37Z" 2024-06-15T09:41:37.429 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:37.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:42.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:42.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:42.432 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:42.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:42.476 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:42.535 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:42.537+0000 7feaf65baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:42.536 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:42.537+0000 7feaf65baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:42.555 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:42.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:42.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:42.559 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:42.560 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:42Z" 2024-06-15T09:41:42.560 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:42.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:47.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:47.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:47.562 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:47.606 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:47.606 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:47.658 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:47.657+0000 7f75a5ad5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:47.659 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:47.661+0000 7f75a5ad5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:47.678 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:47.679 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:47.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:47.681 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:47.682 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:47Z" 2024-06-15T09:41:47.683 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:47.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438536.979596_577.1 2024-06-15T09:41:52.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:52.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:52.686 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:52.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:52.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:52.802 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:52.801+0000 7ff5db1b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:52.803 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:52.805+0000 7ff5db1b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:52.830 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:52.830 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:52.830 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:52.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:52.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:52.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:52.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:52.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:52Z" 2024-06-15T09:41:52.835 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:52.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:41:57.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:57.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:41:57.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:41:57.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:41:57.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:41:57.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:57.941+0000 7f7b78510a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:57.941 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:41:57.941+0000 7f7b78510a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:41:57.972 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:57.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:57.974 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:57.975 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:41:57.976 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:41:57Z" 2024-06-15T09:41:57.977 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:41:57.978 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:02.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:02.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:02.980 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:03.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:03.042 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:03.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:03.092+0000 7fa5d4046a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:03.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:03.092+0000 7fa5d4046a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:03.112 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:03.112 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:03.112 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:03.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:03.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:03.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:03.116 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:03Z" 2024-06-15T09:42:03.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:03.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:08.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:08.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:08.119 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:08.169 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:08.169 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:08.224 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:08.224+0000 7f789082ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:08.225 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:08.224+0000 7f789082ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:08.252 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:08.254 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:08.255 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:08.256 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:08.257 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:08Z" 2024-06-15T09:42:08.258 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:08.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438845.915314_676.1 2024-06-15T09:42:13.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:13.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:13.260 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:13.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:13.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:13.367 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:13.368+0000 7f94dec19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:13.368 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:13.368+0000 7f94dec19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:13.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:13.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:13.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:13.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:13Z" 2024-06-15T09:42:13.390 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:13.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438536.979596_577.1 2024-06-15T09:42:18.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:18.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:18.392 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:18.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:18.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:18.482 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:18.480+0000 7fdeb3bb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:18.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:18.484+0000 7fdeb3bb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:18.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:18.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:18.512 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:18Z" 2024-06-15T09:42:18.513 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:18.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438845.799732_672.1 2024-06-15T09:42:23.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:23.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:23.515 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:23.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:23.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:23.612 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:23.612+0000 7f3d95693a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:23.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:23.612+0000 7f3d95693a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:23.634 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:23.634 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:23.634 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:23.634 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:23.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:23.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:23.638 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:23Z" 2024-06-15T09:42:23.639 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:23.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:28.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:28.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:28.642 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:28.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:28.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:28.727 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:28.728+0000 7fd8d0125a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:28.728 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:28.728+0000 7fd8d0125a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:28.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:28.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:28.749 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:28.750 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:28.751 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:28.752 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:28Z" 2024-06-15T09:42:28.752 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:28.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.076452_581.1 2024-06-15T09:42:33.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:33.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:33.755 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:33.801 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:33.801 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:33.856 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:33.856+0000 7fd375745a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:33.857 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:33.856+0000 7fd375745a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:33.875 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:33.875 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:33.876 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:33.877 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:33.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:33.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:33Z" 2024-06-15T09:42:33.880 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:33.881 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.076452_581.1 2024-06-15T09:42:38.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:38.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:38.883 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:38.926 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:38.926 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:38.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:38.972+0000 7f34abcf1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:38.972 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:38.972+0000 7f34abcf1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:38.991 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:38.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:38.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:38.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:38.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:38.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:38.997 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:38Z" 2024-06-15T09:42:38.997 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:38.998 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:43.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:43.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:43.999 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:44.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:44.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:44.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:44.100+0000 7f803f4aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:44.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:44.100+0000 7f803f4aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.119 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:44.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:44.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:44.123 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:44.124 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:44.124 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:44Z" 2024-06-15T09:42:44.124 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:44.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.076452_581.1 2024-06-15T09:42:49.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:49.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:49.126 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:49.170 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:49.170 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:49.225 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:49.224+0000 7f365d784a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:49.226 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:49.224+0000 7f365d784a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:49.246 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:49.247 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:49.248 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:49.249 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:49.250 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:49.251 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:49Z" 2024-06-15T09:42:49.252 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:49.252 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:54.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:54.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:54.254 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:54.301 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:54.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:54.355+0000 7f86af641a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:54.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:54.355+0000 7f86af641a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:54.383 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:54.383 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:54.383 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:54.383 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:54.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:54.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:54.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:54.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:54.389 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:54Z" 2024-06-15T09:42:54.390 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:54.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:42:59.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:59.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:42:59.392 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:42:59.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:42:59.450 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:42:59.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:59.499+0000 7ff198869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:59.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:42:59.503+0000 7ff198869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:42:59.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:59.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:42:59.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:42:59.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:42:59.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:42:59Z" 2024-06-15T09:42:59.533 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:42:59.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438536.979596_577.1 2024-06-15T09:43:04.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:04.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:04.536 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:04.579 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:04.579 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:04.621 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:04.619+0000 7fd71ea13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:04.622 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:04.619+0000 7fd71ea13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:04.645 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:04.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:04.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:04.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:04.649 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:04.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:04.650 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:04Z" 2024-06-15T09:43:04.650 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:04.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:09.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:09.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:09.652 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:09.695 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:09.695 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:09.739 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:09.739+0000 7f550f874a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:09.740 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:09.739+0000 7f550f874a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:09.760 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:09.760 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:09.761 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:09.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:09.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:09.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:09Z" 2024-06-15T09:43:09.765 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:09.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:14.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:14.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:14.768 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:14.811 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:14.811 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:14.858 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:14.855+0000 7ff0bc136a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:14.858 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:14.859+0000 7ff0bc136a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:14.876 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:14.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:14.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:14.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:14Z" 2024-06-15T09:43:14.881 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:14.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438536.979596_577.1 2024-06-15T09:43:19.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:19.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:19.883 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:19.940 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:19.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:19.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:19.983+0000 7f7cdee09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:19.985 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:19.983+0000 7f7cdee09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:20.003 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:20.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:20.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:20.006 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:19Z" 2024-06-15T09:43:20.008 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:20.009 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:25.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:25.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:25.011 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:25.056 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:25.056 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:25.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:25.099+0000 7f2062dbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:25.102 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:25.099+0000 7f2062dbfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:25.124 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:25.125 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:25.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:25.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:25.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:25.129 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:25Z" 2024-06-15T09:43:25.130 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:25.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:30.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:30.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:30.132 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:30.193 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:30.194 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:30.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:30.255+0000 7f69e1a2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:30.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:30.255+0000 7f69e1a2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:30.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:30.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:30.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:30.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:30.281 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:30Z" 2024-06-15T09:43:30.282 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:30.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:35.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:35.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:35.284 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:35.329 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:35.329 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:35.383 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:35.383+0000 7f16bc4bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:35.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:35.383+0000 7f16bc4bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:35.410 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:35.410 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:35.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:35.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:35.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:35.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:35.415 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:35.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:35.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:35Z" 2024-06-15T09:43:35.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:35.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:40.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:40.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:40.418 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:40.482 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:40.482 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:40.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:40.539+0000 7f20d6260a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:40.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:40.539+0000 7f20d6260a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:40.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:40.559 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:40.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:40.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:40.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:40.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:40.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:40.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:40.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:40.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:40.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:40.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:40.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:40.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:40.570 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:40Z" 2024-06-15T09:43:40.570 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:40.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:45.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:45.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:45.573 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:45.628 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:45.628 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:45.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:45.670+0000 7f258ae84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:45.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:45.670+0000 7f258ae84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:45.689 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:45.689 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:45.690 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:45.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:45.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:45.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:45Z" 2024-06-15T09:43:45.694 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:45.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:50.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:50.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:50.696 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:50.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:50.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:50.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:50.786+0000 7f1acc5dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:50.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:50.790+0000 7f1acc5dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:50.809 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:50.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:50.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:50Z" 2024-06-15T09:43:50.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:50.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:43:55.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:55.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:43:55.817 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:43:55.868 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:43:55.869 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:43:55.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:55.934+0000 7fcfc6471a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:55.939 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:43:55.938+0000 7fcfc6471a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:43:55.958 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:55.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:55.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:43:55.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:43:55Z" 2024-06-15T09:43:55.963 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:43:55.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:00.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:00.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:00.966 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:01.010 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:01.010 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:01.067 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:01.066+0000 7f0b6cf9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:01.068 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:01.066+0000 7f0b6cf9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:01.087 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:01.087 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:01.087 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:01.087 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:01.087 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:01.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:01.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:01.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:01.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:01Z" 2024-06-15T09:44:01.092 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:01.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:06.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:06.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:06.095 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:06.139 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:06.139 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:06.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:06.190+0000 7f83c7d51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:06.194 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:06.194+0000 7f83c7d51a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:06.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:06.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:06.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:06.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:06.221 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:06Z" 2024-06-15T09:44:06.221 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:06.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:11.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:11.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:11.229 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:11.359 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:11.359 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:11.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:11.418+0000 7f423ce8fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:11.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:11.422+0000 7f423ce8fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:11.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:11.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:11.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:11.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:11.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:11Z" 2024-06-15T09:44:11.453 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:11.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:16.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:16.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:16.455 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:16.499 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:16.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:16.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:16.538+0000 7f643c82da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:16.544 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:16.542+0000 7f643c82da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:16.583 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:16.583 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:16.583 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:16.583 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:16.583 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:16.584 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:16.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:16.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:16.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:16.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:16.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:16.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:16.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:16.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:16.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:16.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:16.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:16.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:16.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:16.590 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:16.590 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:16.590 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:16.590 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:16.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:16.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:16.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:16.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:16.595 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:16Z" 2024-06-15T09:44:16.595 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:16.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:21.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:21.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:21.598 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:21.641 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:21.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:21.707 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:21.710+0000 7f2b0a03da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:21.708 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:21.710+0000 7f2b0a03da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:21.726 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:21.727 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:21.730 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:21.731 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:21.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:21.733 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:21Z" 2024-06-15T09:44:21.734 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:21.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:26.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:26.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:26.737 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:26.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:26.780 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:26.843 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:26.842+0000 7f27d984aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:26.844 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:26.846+0000 7f27d984aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:26.862 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:26.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:26.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:26.867 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:26Z" 2024-06-15T09:44:26.868 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:26.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:31.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:31.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:31.871 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:31.917 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:31.917 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:31.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:31.977+0000 7fad2a4b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:31.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:31.977+0000 7fad2a4b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:31.996 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:31.996 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:31.996 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:31.996 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:31.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:31.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:31.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:32.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:31Z" 2024-06-15T09:44:32.001 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:32.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:37.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:37.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:37.003 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:37.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:37.048 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:37.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:37.101+0000 7fa6ebdf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:37.106 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:37.105+0000 7fa6ebdf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:37.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:37.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:37.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:37.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:37.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:37Z" 2024-06-15T09:44:37.140 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:37.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.076452_581.1 2024-06-15T09:44:42.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:42.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:42.143 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:42.188 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:42.188 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:42.238 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:42.237+0000 7f7a34433a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:42.239 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:42.241+0000 7f7a34433a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:42.260 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:42.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:42.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:42.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:42.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:42.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:42.265 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:42Z" 2024-06-15T09:44:42.265 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:42.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:47.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:47.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:47.268 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:47.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:47.318 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:47.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:47.361+0000 7fb7012e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:47.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:47.365+0000 7fb7012e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:47.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:47.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:47.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:47.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:47.390 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:47Z" 2024-06-15T09:44:47.390 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:47.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:52.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:52.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:52.392 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:52.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:52.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:52.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:52.489+0000 7fbab9166a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:52.491 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:52.493+0000 7fbab9166a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.511 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:52.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:52.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:52.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:52Z" 2024-06-15T09:44:52.516 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:52.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:44:57.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:57.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:44:57.518 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:44:57.563 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:44:57.563 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:44:57.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:57.617+0000 7f02a366fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:57.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:44:57.617+0000 7f02a366fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:44:57.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:57.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:57.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:44:57.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:44:57Z" 2024-06-15T09:44:57.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:44:57.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:02.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:02.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:02.646 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:02.691 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:02.691 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:02.750 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:02.749+0000 7fc9927c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:02.751 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:02.753+0000 7fc9927c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:02.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:02.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:02.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:02.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.776 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:02.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:02.777 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:02.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:02.777 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:02Z" 2024-06-15T09:45:02.777 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:02.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:07.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:07.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:07.780 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:07.825 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:07.825 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:07.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:07.881+0000 7fa6912e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:07.880 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:07.881+0000 7fa6912e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:07.905 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:07.905 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:07.905 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:07.906 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:07.906 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:07.907 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:07.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:07.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:07.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:07.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:07.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.114025_584.1", 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.114025Z", 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:07.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:07.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:07.913 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:07.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:07.913 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:07Z" 2024-06-15T09:45:07.913 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:07.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438539.114025_584.1 2024-06-15T09:45:12.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:12.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:12.916 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:12.959 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:12.959 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:13.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:13.021+0000 7fd5a3076a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:13.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:13.021+0000 7fd5a3076a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:13.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:13.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:13.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439520.015783_810.1", 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:18:40.015783Z", 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:13.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:13.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:13.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:13.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:13Z" 2024-06-15T09:45:13.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:13.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718439520.015783_810.1 2024-06-15T09:45:18.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:18.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:18.047 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:18.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:18.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:18.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:18.133+0000 7f0e82f31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:18.135 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:18.137+0000 7f0e82f31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:18.154 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:18.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:18.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:18.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:18.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:18Z" 2024-06-15T09:45:18.159 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:18.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:23.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:23.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:23.162 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:23.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:23.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:23.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:23.252+0000 7f439316da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:23.254 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:23.252+0000 7f439316da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:23.275 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:23.275 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:23.275 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:23.275 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:23.275 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:23.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:23.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:23.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:23.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:23Z" 2024-06-15T09:45:23.280 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:23.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:28.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:28.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:28.282 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:28.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:28.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:28.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:28.384+0000 7f070b192a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:28.385 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:28.384+0000 7f070b192a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.412 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:28.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:28.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:28.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:28.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:28Z" 2024-06-15T09:45:28.418 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:28.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:33.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:33.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:33.420 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:33.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:33.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:33.601 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:33.600+0000 7f5288dd9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:33.601 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:33.600+0000 7f5288dd9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:33.619 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:33.619 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:33.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:33.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:33.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:33.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:33Z" 2024-06-15T09:45:33.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:33.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:38.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:38.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:38.626 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:38.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:38.671 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:38.713 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:38.712+0000 7f6e0be42a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:38.714 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:38.712+0000 7f6e0be42a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:38.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:38.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:38.744 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:38Z" 2024-06-15T09:45:38.744 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:38.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:43.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:43.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:43.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:43.872 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:43.872 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:43.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:43.928+0000 7f1b1e15da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:43.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:43.928+0000 7f1b1e15da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:43.950 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:43.950 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:43.950 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:43.951 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:43.951 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:43.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:43.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:43.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:43.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:43.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:43.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:43.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:43.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:43.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:43.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:43.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:43.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:43.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:43.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:43.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:43.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:43.956 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:43.956 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:43.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:43.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:43.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:43.957 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:43.957 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:43.957 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:43.957 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:43.958 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:43Z" 2024-06-15T09:45:43.962 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:43.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:48.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:48.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:48.966 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:49.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:49.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:49.102 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:49.100+0000 7ff172e79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:49.103 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:49.100+0000 7ff172e79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:49.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:49.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:49.134 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:49.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:49Z" 2024-06-15T09:45:49.136 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:49.138 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:45:54.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:54.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:54.139 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:54.183 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:54.183 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:54.228 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:54.228+0000 7f5c88412a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:54.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:54.228+0000 7f5c88412a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:54.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:54.252 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:54.253 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:54.254 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:54.255 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.256 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:54.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:54.257 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:54.257 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:54.257 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:54Z" 2024-06-15T09:45:54.257 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:54.258 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438536.979596_577.1 2024-06-15T09:45:59.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:59.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:45:59.259 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:45:59.302 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:45:59.302 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:45:59.354 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:59.352+0000 7ff47414ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:59.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:45:59.356+0000 7ff47414ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:45:59.376 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:45:59.377 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:45:59.377 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:45:59.377 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:45:59.377 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:45:59.377 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:59.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:59.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:45:59.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:59.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:45:59.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:45:59.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:45:59.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:45:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:45:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:45:59.386 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:45:59Z" 2024-06-15T09:45:59.386 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:45:59.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444455.726175_1433.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:04.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-06-15T09:46:04.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:04.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:04.446 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:04.501 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:04.502 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:04.655 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:46:04.709 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718444764.441743_1461.1", 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:46:04.441743Z" 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:46:04.710 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:46:04.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:46:04.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:04.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:04.711 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:04.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:04.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:04.869 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:04.868+0000 7f81ab944a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:04.870 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:04.868+0000 7f81ab944a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:04.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:04.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:04.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:04.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:04.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:04Z" 2024-06-15T09:46:04.895 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:04.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:46:09.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:09.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:09.898 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:09.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:09.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:09.992 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:09.992+0000 7f8a15130a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:09.993 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:09.992+0000 7f8a15130a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:10.012 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:10.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:10.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:10.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:10.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:10.017 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:09Z" 2024-06-15T09:46:10.018 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:10.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:15.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:15.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:15.020 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:15.065 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:15.065 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:15.124 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:15.123+0000 7f14669f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:15.125 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:15.123+0000 7f14669f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:15.150 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:15.150 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:15.150 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:15.150 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:15.150 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:15.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:15.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.153 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:15.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:15Z" 2024-06-15T09:46:15.155 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:15.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:20.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:20.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:20.157 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:20.200 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:20.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:20.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:20.275+0000 7f4998b88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:20.288 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:20.287+0000 7f4998b88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:20.324 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:20.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:20.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:20Z" 2024-06-15T09:46:20.329 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:20.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:25.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:25.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:25.332 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:25.375 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:25.375 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:25.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:25.519+0000 7fafa219ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:25.523 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:25.523+0000 7fafa219ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:25.543 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:25.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:25.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:25.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:25Z" 2024-06-15T09:46:25.548 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:25.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:30.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:30.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:30.550 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:30.603 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:30.603 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:30.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:30.643+0000 7f1568ddba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:30.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:30.647+0000 7f1568ddba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:30.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:30.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:30.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:30.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:30.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:30.671 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:30Z" 2024-06-15T09:46:30.672 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:30.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:35.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:35.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:35.674 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:35.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:35.716 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:35.767 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:35.763+0000 7f766ccc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:35.768 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:35.767+0000 7f766ccc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:35.789 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:35.789 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:35.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:35.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:35.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:35.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:35.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:35.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:35Z" 2024-06-15T09:46:35.795 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:35.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:40.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:40.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:40.798 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:40.877 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:40.877 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:40.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:40.923+0000 7f5d18a31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:40.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:40.923+0000 7f5d18a31a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:40.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:40.951 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:40.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:40.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:40.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:40.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:40Z" 2024-06-15T09:46:40.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:40.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:46:45.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:45.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:45.959 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:46.002 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:46.002 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:46.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:46.047+0000 7fd565953a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:46.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:46.051+0000 7fd565953a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:46.071 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:46.071 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:46.071 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:46.072 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:46.072 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:46.072 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:46.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:46.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:46.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:46.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:46.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:46.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:46.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:46.078 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:46Z" 2024-06-15T09:46:46.078 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:46.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:51.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:51.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:51.080 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:51.123 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:51.123 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:51.175 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:51.175+0000 7feb2c6b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:51.176 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:51.175+0000 7feb2c6b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:51.202 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:51.202 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:51.202 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:51.202 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:51.202 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:51.203 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:51.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:51.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:51.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:51Z" 2024-06-15T09:46:51.207 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:51.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:46:56.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:56.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:46:56.209 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:46:56.252 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:46:56.252 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:46:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:56.299+0000 7f8591859a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:46:56.303+0000 7f8591859a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:56.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:56.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:56.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:46:56.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:46:56.326 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:46:56.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:46:56.327 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:46:56Z" 2024-06-15T09:46:56.327 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:46:56.328 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:01.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:01.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:01.329 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:01.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:01.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:01.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:01.423+0000 7fc658a9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:01.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:01.423+0000 7fc658a9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:01.445 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:01.445 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:01.445 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:01.445 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:01.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:01.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:01.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:01.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:01Z" 2024-06-15T09:47:01.450 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:01.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:06.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:06.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:06.452 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:06.570 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:06.570 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:06.635 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:06.634+0000 7fe52b4a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:06.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:06.634+0000 7fe52b4a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.659 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:06.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:06.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:06.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:06Z" 2024-06-15T09:47:06.664 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:06.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:11.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:11.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:11.666 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:11.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:11.710 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:11.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:11.754+0000 7f087e6fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:11.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:11.758+0000 7f087e6fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:11.789 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:11.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:11.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:11.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:11.793 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:11.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:11Z" 2024-06-15T09:47:11.795 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:11.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:47:16.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:16.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:16.797 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:16.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:16.841 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:16.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:16.898+0000 7f27e7dc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:16.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:16.898+0000 7f27e7dc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:16.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:16.924 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:16.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:16.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:16.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:16.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:16Z" 2024-06-15T09:47:16.929 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:16.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:21.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:21.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:21.932 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:21.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:21.975 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:22.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:22.014+0000 7f9952627a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:22.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:22.018+0000 7f9952627a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:22.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:22.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:22.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:22.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:22.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:22.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:22Z" 2024-06-15T09:47:22.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:22.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:27.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:27.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:27.048 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:27.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:27.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:27.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:27.130+0000 7f40d0443a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:27.135 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:27.134+0000 7f40d0443a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:27.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:27.158 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:27Z" 2024-06-15T09:47:27.159 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:27.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:32.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:32.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:32.161 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:32.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:32.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:32.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:32.254+0000 7efcebd19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:32.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:32.254+0000 7efcebd19a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:32.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:32.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:32.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:32.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:32.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:32.281 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:32Z" 2024-06-15T09:47:32.281 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:32.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:37.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:37.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:37.283 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:37.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:37.337 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:37.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:37.386+0000 7f48bd350a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:37.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:37.386+0000 7f48bd350a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:37.410 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:37.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:37.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:37.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:37Z" 2024-06-15T09:47:37.415 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:37.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:42.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:42.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:42.418 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:42.464 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:42.464 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:42.514 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:42.510+0000 7fa548e06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:42.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:42.514+0000 7fa548e06a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:42.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:42.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:42.536 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:42.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:42.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:42.540 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:42Z" 2024-06-15T09:47:42.540 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:42.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:47.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:47.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:47.542 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:47.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:47.587 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:47.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:47.634+0000 7f499e6f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:47.633 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:47.634+0000 7f499e6f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:47.651 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:47.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:47.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:47.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:47.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:47Z" 2024-06-15T09:47:47.656 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:47.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:52.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:52.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:52.659 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:52.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:52.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:52.755 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:52.758+0000 7f7cc3054a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:52.756 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:52.758+0000 7f7cc3054a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:52.778 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:52.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:52.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:52.782 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:52Z" 2024-06-15T09:47:52.783 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:52.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:47:57.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:57.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:47:57.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:47:57.829 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:47:57.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:47:57.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:57.885+0000 7ff380991a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:57.884 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:47:57.885+0000 7ff380991a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:47:57.902 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:57.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:47:57.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:57.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:47:57.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:47:57Z" 2024-06-15T09:47:57.907 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:47:57.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:02.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:02.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:02.910 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:02.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:02.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:03.006 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:03.005+0000 7f39ea396a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:03.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:03.009+0000 7f39ea396a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:03.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:03.030 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:03.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.033 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:03.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:03.034 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:03.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:03.034 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:03Z" 2024-06-15T09:48:03.034 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:03.035 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:08.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:08.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:08.036 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:08.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:08.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:08.123 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:08.125+0000 7fc7cda24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:08.125 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:08.125+0000 7fc7cda24a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.143 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.107383_583.1", 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.107383Z", 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:08.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.147 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:08.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:08.148 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:08.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:08.148 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:08Z" 2024-06-15T09:48:08.148 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:08.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.107383_583.1 2024-06-15T09:48:13.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:13.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:13.150 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:13.194 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:13.194 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:13.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:13.257+0000 7fdb56f74a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:13.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:13.257+0000 7fdb56f74a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:13.277 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:13.277 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:13.277 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:13.277 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:13.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:13.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:13.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:13.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:13Z" 2024-06-15T09:48:13.282 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:13.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:18.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:18.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:18.284 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:18.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:18.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:18.434 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:18.433+0000 7fa8d4a3fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:18.436 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:18.437+0000 7fa8d4a3fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:18.458 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:18.458 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:18.459 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:18.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:18.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:18.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:18.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:18.464 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:18Z" 2024-06-15T09:48:18.464 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:18.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:23.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:23.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:23.466 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:23.510 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:23.511 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:23.564 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:23.565+0000 7f3d8440ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:23.565 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:23.565+0000 7f3d8440ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:23.585 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:23.585 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:23.585 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:23.585 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:23.585 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:23.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:23.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.806995_673.1", 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.806995Z", 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:23.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.589 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:23.590 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:23Z" 2024-06-15T09:48:23.590 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:23.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438845.806995_673.1 2024-06-15T09:48:28.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:28.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:28.593 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:28.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:28.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:28.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:28.685+0000 7fcd8a3bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:28.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:28.685+0000 7fcd8a3bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:28.706 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:28.706 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:28.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:28.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:48:28.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:28.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:28Z" 2024-06-15T09:48:28.711 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:28.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.076452_581.1 2024-06-15T09:48:33.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:33.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:33.713 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:33.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:33.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:33.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:33.821+0000 7f4901c36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:33.823 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:33.825+0000 7f4901c36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:33.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:33.849 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:33.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:33.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:33.852 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:33Z" 2024-06-15T09:48:33.853 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:33.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.076452_581.1 2024-06-15T09:48:38.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:38.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:38.855 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:38.898 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:38.898 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:38.950 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:38.949+0000 7f8c78766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:38.951 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:38.953+0000 7f8c78766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:38.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:38.979 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:38.980 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:38.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:38.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:38.983 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:38Z" 2024-06-15T09:48:38.983 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:38.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:43.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:43.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:43.985 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:44.044 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:44.044 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:44.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:44.112+0000 7fb4f098ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:44.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:44.116+0000 7fb4f098ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:44.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:44.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:44.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:44.142 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:44.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:44.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:44.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:44.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:44.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:44.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:44.146 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:44.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:44.147 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:44Z" 2024-06-15T09:48:44.147 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:44.148 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:49.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:49.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:49.150 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:49.195 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:49.195 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:49.257 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:49.256+0000 7f6213cc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:49.258 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:49.256+0000 7f6213cc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:49.284 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:49.284 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:49.284 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:49.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:49.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T09:48:49.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:49.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:49Z" 2024-06-15T09:48:49.289 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:49.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438538.004281_580.1 2024-06-15T09:48:54.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:54.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:54.295 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:54.338 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:54.338 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:54.381 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:54.380+0000 7fe035e1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:54.382 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:54.380+0000 7fe035e1aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:54.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:54.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:54.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:54.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:54Z" 2024-06-15T09:48:54.409 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:54.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:48:59.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:59.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:48:59.412 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:48:59.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:48:59.456 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:48:59.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:59.520+0000 7f1872678a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:59.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:48:59.520+0000 7f1872678a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:48:59.544 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:59.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.546 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:48:59.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.549 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:48:59.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:48:59.550 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:48:59.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:48:59.550 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:48:59Z" 2024-06-15T09:48:59.550 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:48:59.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:04.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:04.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:04.552 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:04.595 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:04.595 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:04.670 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:04.668+0000 7f3485598a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:04.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:04.672+0000 7f3485598a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:04.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:04.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:04.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:04Z" 2024-06-15T09:49:04.701 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:04.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:09.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:09.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:09.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:09.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:09.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:09.800 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:09.800+0000 7f774c631a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:09.801 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:09.800+0000 7f774c631a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:09.821 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:09.823 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:09.824 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.825 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:09.825 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:09.825 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:09.825 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:09.825 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:09.826 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:09Z" 2024-06-15T09:49:09.827 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:09.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:14.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:14.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:14.830 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:14.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:14.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:14.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:14.928+0000 7f91c29b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:14.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:14.928+0000 7f91c29b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:14.949 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:14.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:14.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:14.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:14.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:14.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:14.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:14Z" 2024-06-15T09:49:14.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:14.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:19.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:19.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:19.960 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:20.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:20.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:20.065 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:20.064+0000 7f1c23a3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:20.066 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:20.064+0000 7f1c23a3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:20.087 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:20.087 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:20.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:20.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:49:20.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:20.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:20Z" 2024-06-15T09:49:20.092 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:20.093 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:49:25.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:25.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:25.095 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:25.137 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:25.138 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:25.186 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:25.184+0000 7f8f722ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:25.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:25.188+0000 7f8f722ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:25.213 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:25.213 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:25.213 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:25.213 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:25.213 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:25.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:25.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:25.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:25.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:25Z" 2024-06-15T09:49:25.218 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:25.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:30.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:30.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:30.220 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:30.263 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:30.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:30.318 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:30.316+0000 7f6695bdfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:30.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:30.320+0000 7f6695bdfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:30.340 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:30.340 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:30.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:30.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:30.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:30.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:30.345 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:30.345 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:30.345 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:30Z" 2024-06-15T09:49:30.345 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:30.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:35.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:35.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:35.347 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:35.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:35.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:35.436 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:35.435+0000 7f250b4c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:35.437 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:35.435+0000 7f250b4c5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:35.457 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:35.458 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:35.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:35.460 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:35.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:35.462 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:35.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:35.463 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:35Z" 2024-06-15T09:49:35.463 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:35.464 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:40.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:40.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:40.465 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:40.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:40.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:40.558 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:40.555+0000 7fb444eb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:40.558 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:40.555+0000 7fb444eb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:40.579 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:40.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:40.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.076452_581.1", 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.076452Z", 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:40.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:40Z" 2024-06-15T09:49:40.584 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:40.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.076452_581.1 2024-06-15T09:49:45.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:45.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:45.587 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:45.630 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:45.630 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:45.677 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:45.675+0000 7f3f920e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:45.678 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:45.675+0000 7f3f920e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:45.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:45.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:45.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:45.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:45.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:45Z" 2024-06-15T09:49:45.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:45.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:49:50.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:50.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:50.705 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:50.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:50.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:50.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:50.787+0000 7f280375ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:50.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:50.791+0000 7f280375ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:50.810 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:50.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:50.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:50.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:50.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:50.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:50Z" 2024-06-15T09:49:50.816 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:50.817 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:49:55.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:55.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:49:55.818 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:49:55.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:49:55.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:49:55.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:55.903+0000 7f4f4f462a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:55.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:49:55.903+0000 7f4f4f462a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:49:55.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:55.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:55.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:55.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:49:55.935 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:49:55Z" 2024-06-15T09:49:55.936 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:49:55.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:00.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:00.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:00.938 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:00.982 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:00.982 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:01.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:01.035+0000 7f8e98ef8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:01.039 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:01.039+0000 7f8e98ef8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:01.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:01.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:01.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438536.979596_577.1", 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:16.979596Z", 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.063 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:01.064 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:01Z" 2024-06-15T09:50:01.064 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:01.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438536.979596_577.1 2024-06-15T09:50:06.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:06.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:06.066 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:06.118 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:06.118 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:06.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:06.159+0000 7fd4f3e5fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:06.163 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:06.163+0000 7fd4f3e5fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:06.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:06.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:06.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:06.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:06.186 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:06.187 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:06Z" 2024-06-15T09:50:06.187 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:06.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:11.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:11.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:11.189 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:11.233 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:11.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:11.273 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:11.271+0000 7fc688d43a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:11.274 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:11.271+0000 7fc688d43a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:11.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:11.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:11.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:11Z" 2024-06-15T09:50:11.303 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:11.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:16.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:16.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:16.306 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:16.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:16.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:16.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:16.399+0000 7f00b9215a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:16.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:16.399+0000 7f00b9215a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:16.420 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:16.420 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:16.420 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:16.420 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:16.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:16.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:16.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:16.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:16Z" 2024-06-15T09:50:16.425 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:16.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:21.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:21.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:21.427 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:21.472 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:21.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:21.519 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:21.519+0000 7f00e9a05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:21.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:21.519+0000 7f00e9a05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:21.540 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:21.540 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:21.540 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:21.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:21.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438538.004281_580.1", 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:18.004281Z", 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:21.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:21Z" 2024-06-15T09:50:21.545 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:21.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438538.004281_580.1 2024-06-15T09:50:26.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:26.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:26.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:26.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:26.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:26.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:26.646+0000 7fb8b7b02a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:26.648 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:26.646+0000 7fb8b7b02a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:26.670 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:26.670 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:26.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:26.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:26.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:26.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:26.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:26.676 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:26Z" 2024-06-15T09:50:26.676 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:26.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:31.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:31.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:31.682 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:31.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:31.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:31.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:31.782+0000 7f22faf9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:31.787 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:31.786+0000 7f22faf9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:31.811 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:31.811 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:31.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:31.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:31.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:31.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:31Z" 2024-06-15T09:50:31.817 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:31.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:36.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:36.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:36.822 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:36.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:36.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:36.922 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:36.918+0000 7fb31432fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:36.922 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:36.922+0000 7fb31432fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:36.942 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:36.943 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:36.944 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438537.989943_579.1", 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:17.989943Z", 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:36.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:36.946 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:36Z" 2024-06-15T09:50:36.947 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:36.948 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438537.989943_579.1 2024-06-15T09:50:41.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:41.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:41.950 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:41.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:41.993 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:42.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:42.050+0000 7fa2b1487a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:42.052 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:42.050+0000 7fa2b1487a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:42.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:42.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:42.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:42.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:42.078 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:42Z" 2024-06-15T09:50:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:42.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.261071_591.1 2024-06-15T09:50:47.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:47.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:47.081 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:47.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:47.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:47.176 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:47.174+0000 7f3abade0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:47.177 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:47.174+0000 7f3abade0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:47.202 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:47.202 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:47.203 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:47.204 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:47.205 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:47.206 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:47Z" 2024-06-15T09:50:47.207 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:47.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.163871_589.1 2024-06-15T09:50:52.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:52.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:52.209 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:52.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:52.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:52.311 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:52.306+0000 7fdd60339a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:52.312 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:52.310+0000 7fdd60339a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:52.336 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:52.337 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:52.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:52.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:52.340 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:52Z" 2024-06-15T09:50:52.341 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:52.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.261071_591.1 2024-06-15T09:50:57.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:57.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:50:57.344 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:50:57.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:50:57.387 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:50:57.463 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:57.462+0000 7f376e5b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:57.464 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:50:57.462+0000 7f376e5b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:50:57.486 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:50:57.486 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:50:57.486 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:57.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:57.488 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:50:57.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:57.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:57.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:50:57.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:50:57Z" 2024-06-15T09:50:57.493 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:50:57.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.152646_587.1 2024-06-15T09:51:02.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:02.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:02.496 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:02.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:02.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:02.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:02.590+0000 7fd107fe5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:02.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:02.590+0000 7fd107fe5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:02.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:02.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:02.614 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:02.615 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:02.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:02.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:02Z" 2024-06-15T09:51:02.618 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:02.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.261071_591.1 2024-06-15T09:51:07.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:07.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:07.620 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:07.664 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:07.665 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:07.732 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:07.734+0000 7f27c10aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:07.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:07.734+0000 7f27c10aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:07.762 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:07.762 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:07.762 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:07.762 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:07.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:07.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:07.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:07Z" 2024-06-15T09:51:07.767 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:07.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718444764.441743_1461.1 target=1_1718438539.261071_591.1 2024-06-15T09:51:12.770 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-06-15T09:51:12.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:12.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:12.819 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:12.862 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:12.862 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:13.017 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:51:13.075 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445072.814929_1495.1", 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:51:12.814929Z" 2024-06-15T09:51:13.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:51:13.077 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:51:13.077 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:13.077 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:51:13.077 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:51:13.077 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:51:13.078 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:51:13.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:13.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:13.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:13.152 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:13.152 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:13.211 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:13.214+0000 7f45c6e5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:13.212 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:13.214+0000 7f45c6e5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:13.233 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:13.234 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:13.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:13.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:13.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:13Z" 2024-06-15T09:51:13.238 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:13.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:51:18.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:18.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:18.240 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:18.306 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:18.306 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:18.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:18.365+0000 7f0498f48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:18.373 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:18.373+0000 7f0498f48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:18.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:18.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:18.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:18.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.426 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:18.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:18.427 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:18.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:18.427 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:18Z" 2024-06-15T09:51:18.427 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:18.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:51:23.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:23.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:23.429 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:23.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:23.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:23.514 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:23.513+0000 7f054247ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:23.515 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:23.517+0000 7f054247ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:23.535 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:23.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:23.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:23.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:23.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:23.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:23.541 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:23Z" 2024-06-15T09:51:23.541 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:23.542 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:51:28.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:28.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:28.543 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:28.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:28.588 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:28.649 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:28.649+0000 7f867ca9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:28.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:28.649+0000 7f867ca9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:28.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:28.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:28.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:28.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:28Z" 2024-06-15T09:51:28.679 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:28.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:51:33.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:33.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:33.683 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:33.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:33.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:33.807 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:33.809+0000 7fdaaf426a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:33.808 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:33.809+0000 7fdaaf426a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:33.825 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:33.826 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:33.826 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:33.826 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:33.826 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:33.826 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:33.827 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:33.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.827 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:33.827 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:33.827 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:33.828 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:33.828 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:33.828 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:33.828 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:33.828 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.829 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:33.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:33.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:33.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:33.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:33.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:33.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:33.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:33.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:33.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:33.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:51:33.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:33.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:33.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:33.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:51:33.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:33.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:33.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:33.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:33.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:33.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:33.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:33.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:33.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:33.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:33.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.837 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:33.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:33.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:33.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:33.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:33Z" 2024-06-15T09:51:33.838 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:33.839 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:51:38.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:38.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:38.841 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:38.884 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:38.885 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:38.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:38.929+0000 7f8b4ba28a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:38.931 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:38.933+0000 7f8b4ba28a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:38.958 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:38.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:38.958 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:38.958 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:38.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:38.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:38.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:38.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:38Z" 2024-06-15T09:51:38.963 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:38.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:51:43.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:43.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:43.966 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:44.008 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:44.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:44.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:44.061+0000 7efcb4f2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:44.063 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:44.065+0000 7efcb4f2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:44.085 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:44.085 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:44.085 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:44.085 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:44.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:44.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:44.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:44.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:44Z" 2024-06-15T09:51:44.090 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:44.091 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:51:49.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:49.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:49.092 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:49.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:49.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:49.286 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:49.285+0000 7f19aa872a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:49.287 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:49.289+0000 7f19aa872a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:49.326 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:49.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:49.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:51:49.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:49.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:49.331 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:49Z" 2024-06-15T09:51:49.332 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:49.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:51:54.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:54.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:54.334 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:54.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:54.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:54.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:54.429+0000 7fa90ce6ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:54.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:54.429+0000 7fa90ce6ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:54.448 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:54.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:54.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:54.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:54.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:54Z" 2024-06-15T09:51:54.453 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:54.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:51:59.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:59.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:51:59.455 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:51:59.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:51:59.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:51:59.551 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:59.553+0000 7f8a5dfc0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:59.552 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:51:59.553+0000 7f8a5dfc0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:51:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:51:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:51:59.577 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:51:59Z" 2024-06-15T09:51:59.578 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:51:59.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:04.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:04.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:04.580 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:04.637 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:04.637 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:04.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:04.681+0000 7efcabe5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:04.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:04.681+0000 7efcabe5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:04.700 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:04.700 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:04.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:04.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:04.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:04.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:04Z" 2024-06-15T09:52:04.705 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:04.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:09.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:09.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:09.708 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:09.752 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:09.752 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:09.807 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:09.808+0000 7f8e16728a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:09.808 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:09.808+0000 7f8e16728a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:09.829 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:09.829 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:09.829 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:09.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.762966_670.1", 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.762966Z", 2024-06-15T09:52:09.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:09.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:09Z" 2024-06-15T09:52:09.834 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:09.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.762966_670.1 2024-06-15T09:52:14.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:14.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:14.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:14.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:14.880 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:14.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:14.924+0000 7f0395053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:14.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:14.924+0000 7f0395053a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:14.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:14.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:14.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:14.951 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:14Z" 2024-06-15T09:52:14.952 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:14.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:19.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:19.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:19.955 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:20.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:20.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:20.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:20.072+0000 7f1a6b011a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:20.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:20.072+0000 7f1a6b011a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:20.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:20.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:20.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:20.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.098 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:20.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:20.099 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:20.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:20.099 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:20Z" 2024-06-15T09:52:20.099 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:20.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:25.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:25.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:25.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:25.145 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:25.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:25.205 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:25.204+0000 7f50877d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:25.206 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:25.204+0000 7f50877d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:25.232 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:25.232 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:25.233 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:25.234 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:25.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:25.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.237 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:25.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:25.238 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:25.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:25.238 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:25Z" 2024-06-15T09:52:25.238 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:25.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:30.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:30.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:30.241 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:30.284 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:30.284 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:30.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:30.340+0000 7fcb53bf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:30.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:30.340+0000 7fcb53bf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:30.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:30.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:30.372 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:30.373 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:30.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:30.374 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:30Z" 2024-06-15T09:52:30.374 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:30.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:52:35.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:35.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:35.376 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:35.432 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:35.432 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:35.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:35.484+0000 7f780ee9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:35.487 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:35.488+0000 7f780ee9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:35.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:35.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:35.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:35.512 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:35Z" 2024-06-15T09:52:35.513 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:35.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:40.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:40.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:40.515 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:40.559 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:40.559 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:40.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:40.616+0000 7f372a904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:40.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:40.620+0000 7f372a904a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:40.642 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:40.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:40.644 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:40.645 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:40.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:40Z" 2024-06-15T09:52:40.647 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:40.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:45.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:45.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:45.649 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:45.693 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:45.694 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:45.751 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:45.752+0000 7fcbd1ce1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:45.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:45.752+0000 7fcbd1ce1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:45.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:45.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:45.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:45.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:45Z" 2024-06-15T09:52:45.782 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:45.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:52:50.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:50.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:50.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:50.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:50.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:50.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:50.892+0000 7fbc44e8ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:50.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:50.892+0000 7fbc44e8ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:50.919 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:50.919 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:50.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:50.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:50.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:50.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:50.924 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:50Z" 2024-06-15T09:52:50.925 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:50.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:52:55.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:55.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:52:55.927 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:52:55.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:52:55.973 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:52:56.035 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:56.036+0000 7fa0ca465a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:56.036 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:52:56.036+0000 7fa0ca465a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:52:56.061 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:52:56.061 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:52:56.061 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:52:56.061 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:52:56.062 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:52:56.062 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:52:56.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:56.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:56.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:52:56.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.066 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:52:56.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:52:56.067 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:52:56.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:52:56.067 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:52:56Z" 2024-06-15T09:52:56.067 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:52:56.068 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:53:01.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:01.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:01.069 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:01.121 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:01.121 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:01.166 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:01.163+0000 7ff4e9521a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:01.167 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:01.167+0000 7ff4e9521a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:01.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:01.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:01.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:01.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:01Z" 2024-06-15T09:53:01.190 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:01.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:53:06.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:06.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:06.193 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:06.237 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:06.237 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:06.284 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:06.283+0000 7fab984cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:06.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:06.283+0000 7fab984cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:06.303 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:06.303 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:06.303 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:06.303 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:06.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:06.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:06Z" 2024-06-15T09:53:06.308 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:06.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.179989_590.1 2024-06-15T09:53:11.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:11.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:11.311 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:11.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:11.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:11.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:11.399+0000 7efd3540aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:11.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:11.403+0000 7efd3540aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:11.421 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:11.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:11Z" 2024-06-15T09:53:11.426 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:11.427 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.799732_672.1 2024-06-15T09:53:16.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:16.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:16.429 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:16.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:16.473 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:16.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:16.535+0000 7f96f5a95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:16.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:16.535+0000 7f96f5a95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.558 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:16.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:16.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:16.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:16.563 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:16Z" 2024-06-15T09:53:16.563 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:16.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:53:21.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:21.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:21.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:21.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:21.610 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:21.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:21.659+0000 7f60beccda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:21.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:21.663+0000 7f60beccda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:21.685 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:21.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:21.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:21.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:21.691 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:21Z" 2024-06-15T09:53:21.691 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:21.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:53:26.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:26.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:26.693 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:26.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:26.736 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:26.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:26.787+0000 7f8369258a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:26.791 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:26.791+0000 7f8369258a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:26.813 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:26.813 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:26.814 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:26.815 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:26.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:26.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:26Z" 2024-06-15T09:53:26.818 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:26.819 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.799732_672.1 2024-06-15T09:53:31.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:31.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:31.820 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:31.864 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:31.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:31.920 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:31.919+0000 7f3bb7675a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:31.921 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:31.919+0000 7f3bb7675a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:31.942 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:31.942 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:31.943 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:31.943 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:31.943 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:31.943 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:31.943 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:31.944 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:31.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:31.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:31.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:31Z" 2024-06-15T09:53:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:31.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:53:36.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:36.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:36.951 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:36.995 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:36.996 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:37.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:37.043+0000 7f559bf88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:37.046 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:37.043+0000 7f559bf88a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:37.071 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:37.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:37.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:37.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:37.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:37Z" 2024-06-15T09:53:37.076 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:37.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:53:42.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:42.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:42.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:42.121 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:42.121 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:42.171 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:42.171+0000 7ff619ee1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:42.172 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:42.171+0000 7ff619ee1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:42.190 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:42.190 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:42.190 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:42.190 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:42.190 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:42.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:42.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.194 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:42.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:42.195 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:42.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:42.195 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:42Z" 2024-06-15T09:53:42.195 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:42.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:53:47.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:47.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:47.197 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:47.242 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:47.242 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:47.284 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:47.283+0000 7fbdc7de7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:47.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:47.283+0000 7fbdc7de7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:47.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:47.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:53:47.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:47.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:47Z" 2024-06-15T09:53:47.308 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:47.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:53:52.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:52.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:52.311 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:52.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:52.355 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:52.397 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:52.394+0000 7f5443a7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:52.397 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:52.394+0000 7f5443a7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:52.419 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:52.420 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:52.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:52.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:52.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:52Z" 2024-06-15T09:53:52.425 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:52.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:53:57.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:57.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:53:57.427 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:53:57.482 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:53:57.482 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:53:57.536 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:57.534+0000 7fdc9e8ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:57.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:53:57.534+0000 7fdc9e8ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:53:57.555 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:53:57.555 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:57.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:53:57.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:53:57Z" 2024-06-15T09:53:57.560 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:53:57.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:54:02.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:02.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:02.563 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:02.606 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:02.607 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:02.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:02.650+0000 7ffb1a2a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:02.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:02.654+0000 7ffb1a2a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.676 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:02.678 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:02.679 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:02.680 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:02.681 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:02.681 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:02Z" 2024-06-15T09:54:02.681 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:02.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.179989_590.1 2024-06-15T09:54:07.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:07.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:07.683 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:07.725 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:07.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:07.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:07.770+0000 7f2fa281ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:07.775 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:07.774+0000 7f2fa281ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:07.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:07.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:07.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.800 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:07.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:07.801 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:07.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:07.801 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:07Z" 2024-06-15T09:54:07.801 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:07.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:54:12.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:12.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:12.804 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:12.847 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:12.847 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:12.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:12.898+0000 7fcad2932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:12.903 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:12.902+0000 7fcad2932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:12.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:12.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:12.924 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:12.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:12.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:54:12.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:12.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:12Z" 2024-06-15T09:54:12.929 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:12.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:54:17.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:17.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:17.931 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:17.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:17.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:18.018 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:18.014+0000 7f39f4a4fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:18.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:18.018+0000 7f39f4a4fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:18.037 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:18.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:18.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:18.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:18.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:18.042 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:18Z" 2024-06-15T09:54:18.042 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:18.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:54:23.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:23.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:23.044 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:23.086 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:23.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:23.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:23.134+0000 7fbc05933a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:23.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:23.134+0000 7fbc05933a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:23.155 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:23.155 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:23.155 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:23.155 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:23Z" 2024-06-15T09:54:23.160 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:23.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:54:28.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:28.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:28.162 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:28.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:28.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:28.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:28.250+0000 7fb2227f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:28.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:28.250+0000 7fb2227f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:28.272 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:28.272 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:28.272 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:28.272 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:28.272 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:28.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:28.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:28.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:28.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:28Z" 2024-06-15T09:54:28.277 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:28.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:54:33.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:33.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:33.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:33.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:33.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:33.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:33.374+0000 7f37db60ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:33.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:33.374+0000 7f37db60ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:33Z" 2024-06-15T09:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:33.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:54:38.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:38.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:38.400 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:38.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:38.444 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:38.499 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:38.502+0000 7fb75896fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:38.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:38.502+0000 7fb75896fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:38.519 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:38.519 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:38.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:38.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:38.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:38Z" 2024-06-15T09:54:38.524 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:38.525 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.266881_592.1 2024-06-15T09:54:43.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:43.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:43.526 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:43.572 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:43.572 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:43.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:43.637+0000 7f7116c3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:43.637 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:43.637+0000 7f7116c3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:43.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:43.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:54:43.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:43.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:43Z" 2024-06-15T09:54:43.662 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:43.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:54:48.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:48.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:48.664 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:48.707 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:48.707 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:48.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:48.753+0000 7ff1aa64aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:48.753 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:48.753+0000 7ff1aa64aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.778 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:48.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:48.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:48.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:48Z" 2024-06-15T09:54:48.782 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:48.783 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:54:53.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:53.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:53.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:53.829 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:53.829 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:53.878 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:53.877+0000 7fa35681ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:53.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:53.881+0000 7fa35681ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:53.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:53.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:53.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:53.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:53.909 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:53.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:53.909 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:53Z" 2024-06-15T09:54:53.909 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:53.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:54:58.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:58.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:54:58.911 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:54:58.954 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:54:58.954 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:54:59.007 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:59.009+0000 7fbeeda5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:59.008 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:54:59.009+0000 7fbeeda5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:54:59.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:59.030 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:54:59.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:59.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:54:59.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:54:59.034 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:54:59Z" 2024-06-15T09:54:59.035 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:54:59.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:55:04.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:04.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:04.037 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:04.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:04.091 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:04.200 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:04.201+0000 7f2413dd9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:04.201 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:04.201+0000 7f2413dd9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:04.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:04.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:04.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:04.229 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:04.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:04.229 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:04Z" 2024-06-15T09:55:04.229 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:04.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.915314_676.1 2024-06-15T09:55:09.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:09.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:09.231 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:09.274 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:09.274 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:09.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:09.321+0000 7fa9d512aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:09.323 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:09.325+0000 7fa9d512aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.348 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:09.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:09.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:09.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:09.353 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:09Z" 2024-06-15T09:55:09.353 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:09.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.268091_593.1 2024-06-15T09:55:14.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:14.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:14.355 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:14.397 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:14.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:14.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:14.441+0000 7f8c51d5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:14.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:14.441+0000 7f8c51d5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:14.461 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:14.461 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:14.461 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:14.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:14.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:14.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:14.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:14Z" 2024-06-15T09:55:14.466 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:14.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:55:19.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:19.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:19.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:19.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:19.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:19.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:19.617+0000 7f74b2a34a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:19.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:19.617+0000 7f74b2a34a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:19.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:19.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:19.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:19.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:19Z" 2024-06-15T09:55:19.641 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:19.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:55:24.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:24.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:24.644 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:24.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:24.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:24.738 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:24.737+0000 7f5e786f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:24.739 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:24.741+0000 7f5e786f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:24.758 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.759 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:24.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:24.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:24.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:24.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:24.764 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:24Z" 2024-06-15T09:55:24.765 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:24.766 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.752631_668.1 2024-06-15T09:55:29.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:29.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:29.767 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:29.810 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:29.810 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:29.861 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:29.861+0000 7f1908096a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:29.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:29.861+0000 7f1908096a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:29.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:29.882 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:29.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:29.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:29Z" 2024-06-15T09:55:29.886 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:29.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:55:34.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:34.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:34.888 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:34.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:34.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:34.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:34.984+0000 7f40a9fd3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:34.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:34.984+0000 7f40a9fd3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:35.009 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:35.010 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:35.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:35.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:35.014 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:35.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:35.014 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:34Z" 2024-06-15T09:55:35.014 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:35.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:55:40.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:40.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:40.017 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:40.060 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:40.061 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:40.107 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:40.108+0000 7fc9abc62a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:40.108 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:40.108+0000 7fc9abc62a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:40.134 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:40.134 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:40.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:40.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:40.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:40Z" 2024-06-15T09:55:40.140 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:40.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:55:45.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:45.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:45.143 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:45.186 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:45.186 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:45.262 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:45.260+0000 7f9d544b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:45.263 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:45.264+0000 7f9d544b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:45.283 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:45.283 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:45.283 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:45.283 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:45.284 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:45.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:45.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:45.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:45Z" 2024-06-15T09:55:45.288 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:45.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.163871_589.1 2024-06-15T09:55:50.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:50.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:50.291 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:50.334 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:50.334 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:50.385 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:50.384+0000 7f02f9405a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:50.386 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:50.384+0000 7f02f9405a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:50.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:50.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:50.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:50.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:50.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:50.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:50Z" 2024-06-15T09:55:50.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:50.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:55:55.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:55.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:55:55.419 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:55:55.461 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:55:55.461 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:55:55.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:55.504+0000 7f60c126ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:55.505 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:55:55.504+0000 7f60c126ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:55:55.526 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:55:55.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:55:55.526 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:55:55.527 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:55:55.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:55.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:55:55.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:55:55Z" 2024-06-15T09:55:55.531 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:55:55.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.152646_587.1 2024-06-15T09:56:00.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:00.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:00.534 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:00.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:00.582 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:00.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:00.628+0000 7f3d7dfc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:00.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:00.632+0000 7f3d7dfc8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:00.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:00.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:00.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.973139_679.1", 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.973139Z", 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:00.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:00.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:00Z" 2024-06-15T09:56:00.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:00.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438845.973139_679.1 2024-06-15T09:56:05.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:05.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:05.659 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:05.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:05.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:05.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:05.748+0000 7f8a58846a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:05.748 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:05.748+0000 7f8a58846a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:05.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:05.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:05.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:05.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:05Z" 2024-06-15T09:56:05.773 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:05.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:56:10.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:10.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:10.776 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:10.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:10.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:10.866 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:10.864+0000 7f7c4ba77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:10.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:10.868+0000 7f7c4ba77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:10.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:10.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:10.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:10.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:10.895 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:10Z" 2024-06-15T09:56:10.895 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:10.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:56:15.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:15.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:15.897 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:15.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:15.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:16.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:16.004+0000 7eff9098da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:16.005 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:16.004+0000 7eff9098da40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:16.024 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:16.024 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:16.024 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:16.024 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:16.024 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:16.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:16.026 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:16.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.028 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:16.029 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:16.029 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:16.029 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:16.029 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:16Z" 2024-06-15T09:56:16.029 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:16.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445072.814929_1495.1 target=1_1718438539.261071_591.1 2024-06-15T09:56:21.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-06-15T09:56:21.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:21.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:21.088 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:21.153 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:21.153 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:21.311 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:56:21.340 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T09:56:21.341 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T09:56:21.342 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T09:56:21.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:21.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:21.343 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:21.384 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:21.384 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:21.427 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:21.428+0000 7fe7746aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:21.428 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:21.428+0000 7fe7746aca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.447 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:21.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:21.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:21.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:21.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:21.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:21.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:21.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:21.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:21.453 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:21Z" 2024-06-15T09:56:21.453 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:21.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T09:56:26.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:26.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:26.455 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:26.499 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:26.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:26.540 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:26.539+0000 7f9f6b13ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:26.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:26.539+0000 7f9f6b13ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:26.566 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:26.566 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:26.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:26.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:26.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:26.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:26Z" 2024-06-15T09:56:26.571 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:26.572 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:56:31.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:31.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:31.574 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:31.622 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:31.622 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:31.663 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:31.663+0000 7fdb45b17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:31.664 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:31.663+0000 7fdb45b17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:31.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:31.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:31.683 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:31.683 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:31.683 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:31.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:31.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:31.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:31.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:31.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:31.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:31.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:31.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:31.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:31.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:31.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:31.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:31.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:31.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:31.690 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:56:31.690 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:31.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:31.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:31.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:31.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:31.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:31.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:31.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:31.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:31.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:31.694 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:31Z" 2024-06-15T09:56:31.694 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:31.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:56:36.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:36.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:36.697 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:36.741 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:36.741 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:36.788 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:36.787+0000 7f2b828cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:36.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:36.787+0000 7f2b828cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:36.809 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:36.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:36.809 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:36.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:36.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:36.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:56:36.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:36.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:36Z" 2024-06-15T09:56:36.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:36.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:56:41.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:41.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:41.816 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:41.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:41.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:41.906 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:41.903+0000 7f33935cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:41.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:41.907+0000 7f33935cfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:41.928 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:41.928 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:41.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:41.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:56:41.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:41.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:41Z" 2024-06-15T09:56:41.933 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:41.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T09:56:46.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:46.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:46.936 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:46.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:46.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:47.029 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:47.027+0000 7fecec32ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:47.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:47.027+0000 7fecec32ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:47.050 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:47.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:47.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:47.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.054 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:47.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:47Z" 2024-06-15T09:56:47.056 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:47.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T09:56:52.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:52.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:52.058 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:52.102 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:52.102 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:52.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:52.151+0000 7f7da9a63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:52.154 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:52.151+0000 7f7da9a63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:52.172 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:52.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:52.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:52.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:52.176 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:52Z" 2024-06-15T09:56:52.177 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:52.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:56:57.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:57.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:56:57.180 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:56:57.226 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:56:57.226 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:56:57.282 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:57.279+0000 7f1c3396aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:57.283 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:56:57.283+0000 7f1c3396aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:56:57.309 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:56:57.309 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:56:57.309 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:57.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:57.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:56:57.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:57.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:56:57Z" 2024-06-15T09:56:57.314 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:56:57.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:57:02.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:02.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:02.317 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:02.361 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:02.361 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:02.421 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:02.419+0000 7f802e92aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:02.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:02.419+0000 7f802e92aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:02.449 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:02.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:02.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:02.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:02.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:02Z" 2024-06-15T09:57:02.454 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:02.455 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:57:07.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:07.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:07.457 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:07.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:07.501 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:07.553 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:07.551+0000 7f2cea524a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:07.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:07.551+0000 7f2cea524a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:07.572 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:07.572 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:07.573 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:07.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:07.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:07.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:07Z" 2024-06-15T09:57:07.577 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:07.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:57:12.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:12.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:12.581 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:12.624 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:12.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:12.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:12.683+0000 7f1bcf7bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:12.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:12.683+0000 7f1bcf7bea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:12.712 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:12.712 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:12.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:12.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.760139_669.1", 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.760139Z", 2024-06-15T09:57:12.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:12.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:12.717 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:12.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:12.718 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:12Z" 2024-06-15T09:57:12.718 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:12.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.760139_669.1 2024-06-15T09:57:17.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:17.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:17.720 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:17.778 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:17.778 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:17.833 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:17.830+0000 7f27e9d63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:17.834 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:17.830+0000 7f27e9d63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:17.858 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:17.859 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:17.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:17.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:17.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.863 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:17.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:17Z" 2024-06-15T09:57:17.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:17.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:57:22.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:22.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:22.867 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:22.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:22.910 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:22.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:22.950+0000 7fe26cc9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:22.953 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:22.950+0000 7fe26cc9aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:22.974 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:22.975 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:22.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:57:22.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:22.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:22Z" 2024-06-15T09:57:22.979 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:22.980 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:57:27.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:27.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:27.982 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:28.022 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:28.022 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:28.071 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:28.066+0000 7f336f691a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:28.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:28.070+0000 7f336f691a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.094 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:28.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:28.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:28.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:28Z" 2024-06-15T09:57:28.099 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:28.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T09:57:33.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:33.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:33.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:33.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:33.147 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:33.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:33.194+0000 7fb2aa8f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:33.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:33.194+0000 7fb2aa8f0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:33.219 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:33.219 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:33.219 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:33.219 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:33.219 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:33.220 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:33.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:33.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:33.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:33.225 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:33.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:33.226 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:33Z" 2024-06-15T09:57:33.226 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:33.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:57:38.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:38.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:38.228 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:38.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:38.273 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:38.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:38.318+0000 7fad30886a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:38.320 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:38.318+0000 7fad30886a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:38.339 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:38.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:38.339 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:38.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:38.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:38.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:38.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:38Z" 2024-06-15T09:57:38.344 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:38.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:57:43.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:43.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:43.347 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:43.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:43.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:43.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:43.430+0000 7fe9bfbb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:43.432 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:43.430+0000 7fe9bfbb8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:43.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:43.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:43.455 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:43.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:43.457 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:43Z" 2024-06-15T09:57:43.458 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:43.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:57:48.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:48.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:48.460 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:48.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:48.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:48.567 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:48.562+0000 7fe8d4890a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:48.568 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:48.566+0000 7fe8d4890a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:48.595 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:48.595 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:48.595 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:48.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:48.597 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:48.598 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:48.599 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:48Z" 2024-06-15T09:57:48.600 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:48.601 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:57:53.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:53.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:53.603 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:53.646 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:53.646 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:53.688 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:53.686+0000 7f4792be9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:53.689 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:53.686+0000 7f4792be9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:53.707 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:53.708 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:53.708 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:53.708 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:53.708 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:53.708 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:53.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:53.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:53.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:53.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:53.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:53.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:53.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:53.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:53.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:53.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:53.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:53.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.769917_671.1", 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.769917Z", 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:53.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:53.716 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:53Z" 2024-06-15T09:57:53.717 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:53.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.769917_671.1 2024-06-15T09:57:58.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:58.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:57:58.719 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:57:58.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:57:58.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:57:58.816 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:58.818+0000 7f9fff87ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:58.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:57:58.818+0000 7f9fff87ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:57:58.837 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:57:58.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.839 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:58.840 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:57:58.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.842 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:57:58.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:57:58.843 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:57:58.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:57:58.843 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:57:58Z" 2024-06-15T09:57:58.843 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:57:58.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:58:03.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:03.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:03.846 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:03.889 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:03.889 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:03.941 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:03.942+0000 7f71d9265a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:03.944 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:03.946+0000 7f71d9265a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:03.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:03.963 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:03.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:03.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:03.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:03.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:03Z" 2024-06-15T09:58:03.968 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:03.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:58:08.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:08.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:08.970 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:09.025 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:09.025 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:09.086 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:09.085+0000 7f9557097a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:09.087 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:09.089+0000 7f9557097a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:09.104 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:09.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:09.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:09.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.109 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:09.110 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:09Z" 2024-06-15T09:58:09.110 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:09.111 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T09:58:14.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:14.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:14.112 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:14.155 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:14.156 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:14.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:14.197+0000 7f48492b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:14.197 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:14.197+0000 7f48492b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:14.215 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:14.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:14.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:14.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:14Z" 2024-06-15T09:58:14.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:14.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T09:58:19.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:19.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:19.222 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:19.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:19.265 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:19.308 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:19.309+0000 7fbc2deb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:19.309 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:19.309+0000 7fbc2deb4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:19.329 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:19.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:19.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.332 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.333 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:19.334 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:19.334 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:19.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:19.334 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:19Z" 2024-06-15T09:58:19.334 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:19.335 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:58:24.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:24.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:24.336 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:24.397 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:24.397 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:24.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:24.445+0000 7f5e2aaafa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:24.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:24.445+0000 7f5e2aaafa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.469 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:24.470 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.471 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.762966_670.1", 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.762966Z", 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:24.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:24.474 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:24Z" 2024-06-15T09:58:24.475 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:24.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.762966_670.1 2024-06-15T09:58:29.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:29.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:29.478 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:29.520 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:29.520 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:29.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:29.569+0000 7f07164eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:29.571 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:29.573+0000 7f07164eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:29.591 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:29.591 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:29.591 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:29.591 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:29.592 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:29.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:58:29.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:29.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:29Z" 2024-06-15T09:58:29.596 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:29.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:58:34.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:34.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:34.598 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:34.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:34.643 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:34.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:34.685+0000 7fa3dc2a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:34.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:34.685+0000 7fa3dc2a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:34.706 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:34.707 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.708 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:34.709 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:34.710 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:34Z" 2024-06-15T09:58:34.711 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:34.711 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:58:39.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:39.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:39.713 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:39.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:39.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:39.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:39.817+0000 7f43d6c43a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:39.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:39.817+0000 7f43d6c43a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:39.841 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:39.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:39.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:39.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:39.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:39.846 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:39.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:39.847 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:39Z" 2024-06-15T09:58:39.847 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:39.848 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:58:44.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:44.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:44.850 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:44.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:44.894 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:44.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:44.937+0000 7fdbf5811a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:44.937 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:44.937+0000 7fdbf5811a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:44.956 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:44.956 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:44.956 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:44.956 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:44.957 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:44.958 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:58:44.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:44.960 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:44Z" 2024-06-15T09:58:44.961 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:44.962 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:58:49.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:49.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:49.963 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:50.008 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:50.008 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:50.066 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:50.065+0000 7f786ec9fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:50.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:50.069+0000 7f786ec9fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:50.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:50.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:50.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:50.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:50.097 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:50Z" 2024-06-15T09:58:50.097 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:50.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T09:58:55.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:55.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:58:55.100 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:58:55.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:58:55.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:58:55.235 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:55.237+0000 7fb985dc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:55.236 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:58:55.237+0000 7fb985dc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:58:55.265 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:58:55.265 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:58:55.265 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:55.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:55.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:55.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:58:55.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:58:55Z" 2024-06-15T09:58:55.270 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:58:55.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T09:59:00.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:00.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:00.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:00.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:00.315 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:00.358 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:00.356+0000 7f175c01aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:00.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:00.360+0000 7f175c01aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:00.379 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:00.379 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:00.379 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:00.379 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:00.379 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:00.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:00.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:00.382 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:00.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:00Z" 2024-06-15T09:59:00.384 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:00.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:05.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:05.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:05.386 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:05.430 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:05.430 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:05.482 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:05.480+0000 7f509018aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:05.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:05.484+0000 7f509018aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:05.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:05.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:05.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:05.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:05.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:05.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:05.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:05Z" 2024-06-15T09:59:05.513 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:05.514 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:10.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:10.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:10.516 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:10.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:10.559 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:10.640 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:10.640+0000 7ff933acfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:10.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:10.640+0000 7ff933acfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:10.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:10.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:10.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:10.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:10.669 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:10.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:10.669 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:10Z" 2024-06-15T09:59:10.669 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:10.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:59:15.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:15.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:15.671 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:15.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:15.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:15.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:15.760+0000 7f418d56ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:15.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:15.760+0000 7f418d56ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:15.782 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:15.782 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:15.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:15.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:15.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:15.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:15Z" 2024-06-15T09:59:15.787 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:15.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T09:59:20.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:20.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:20.790 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:20.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:20.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:20.879 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:20.880+0000 7f21780e1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:20.880 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:20.880+0000 7f21780e1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:20.898 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:20.898 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:20.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:20.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T09:59:20.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:20.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:20Z" 2024-06-15T09:59:20.903 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:20.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T09:59:25.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:25.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:25.906 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:25.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:25.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:26.002 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:26.000+0000 7f896bb58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:26.003 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:26.004+0000 7f896bb58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:26.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:26.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.027 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.028 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:26.029 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:26Z" 2024-06-15T09:59:26.030 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:26.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:31.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:31.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:31.032 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:31.076 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:31.076 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:31.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:31.128+0000 7f2b92c70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:31.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:31.128+0000 7f2b92c70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:31.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:31.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:31.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:31.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:31Z" 2024-06-15T09:59:31.153 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:31.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:36.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:36.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:36.156 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:36.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:36.201 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:36.254 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:36.252+0000 7f75f96eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:36.255 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:36.256+0000 7f75f96eda40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:36.275 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:36.275 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:36.275 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:36.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:36.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:36.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:36.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:36Z" 2024-06-15T09:59:36.280 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:36.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:41.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:41.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:41.283 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:41.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:41.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:41.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:41.380+0000 7fc8b727fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:41.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:41.380+0000 7fc8b727fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:41.401 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:41.401 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:41.402 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:41.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:41.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:41.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:41Z" 2024-06-15T09:59:41.406 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:41.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T09:59:46.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:46.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:46.408 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:46.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:46.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:46.504 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:46.504+0000 7f70008e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:46.505 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:46.504+0000 7f70008e9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:46.525 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:46.525 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:46.525 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:46.525 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:46.525 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:46.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:46.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:46.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:46.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:46Z" 2024-06-15T09:59:46.530 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:46.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:59:51.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:51.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:51.532 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:51.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:51.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:51.633 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:51.631+0000 7f39df24ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:51.634 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:51.631+0000 7f39df24ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:51.653 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:51.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:51.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:51.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:51.658 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:51.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:51.658 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:51Z" 2024-06-15T09:59:51.658 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:51.659 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T09:59:56.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:56.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T09:59:56.660 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T09:59:56.703 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T09:59:56.704 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T09:59:56.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:56.743+0000 7f15c2547a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:56.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T09:59:56.747+0000 7f15c2547a40 20 RGW-SYNC:meta: read sync status 2024-06-15T09:59:56.768 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T09:59:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T09:59:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T09:59:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440137.019720_911.1", 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:57.019720Z", 2024-06-15T09:59:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T09:59:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T09:59:56Z" 2024-06-15T09:59:56.773 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T09:59:56.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440137.019720_911.1 2024-06-15T10:00:01.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:01.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:01.776 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:01.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:01.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:01.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:01.875+0000 7fd4e8f2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:01.876 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:01.875+0000 7fd4e8f2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:01.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:01.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:01.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:01Z" 2024-06-15T10:00:01.901 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:01.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:06.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:06.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:06.903 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:06.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:06.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:07.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:07.003+0000 7fdbf7057a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:07.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:07.003+0000 7fdbf7057a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:07.030 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:07.030 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:07Z" 2024-06-15T10:00:07.035 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:07.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:12.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:12.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:12.038 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:12.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:12.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:12.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:12.123+0000 7f9af183aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:12.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:12.127+0000 7f9af183aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:12.146 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:12Z" 2024-06-15T10:00:12.152 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:12.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:17.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:17.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:17.154 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:17.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:17.206 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:17.266 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:17.263+0000 7fdda9545a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:17.267 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:17.267+0000 7fdda9545a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:17.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:17.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:17.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:17.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:17Z" 2024-06-15T10:00:17.289 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:17.290 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:22.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:22.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:22.292 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:22.335 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:22.335 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:22.379 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:22.379+0000 7fafcfe53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:22.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:22.379+0000 7fafcfe53a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:22.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:22.401 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:22.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:22.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:22.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:00:22.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:22.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:22.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:22.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:22Z" 2024-06-15T10:00:22.409 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:22.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:00:27.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:27.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:27.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:27.454 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:27.454 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:27.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:27.499+0000 7fd0972a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:27.502 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:27.499+0000 7fd0972a0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:27.520 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:27Z" 2024-06-15T10:00:27.527 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:27.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:00:32.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:32.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:32.529 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:32.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:32.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:32.643 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:32.643+0000 7f68aa622a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:32.644 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:32.643+0000 7f68aa622a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:32.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:32.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:32.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.769917_671.1", 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.769917Z", 2024-06-15T10:00:32.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:32.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:32Z" 2024-06-15T10:00:32.673 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:32.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.769917_671.1 2024-06-15T10:00:37.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:37.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:37.676 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:37.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:37.723 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:37.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:37.783+0000 7f2c8b5a8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:37.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:37.783+0000 7f2c8b5a8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:37.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:37.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:37.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:37.813 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:37.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:37.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:37Z" 2024-06-15T10:00:37.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:37.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:42.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:42.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:42.816 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:42.860 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:42.861 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:42.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:42.922+0000 7f11e1622a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:42.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:42.922+0000 7f11e1622a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:42.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:42.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:42.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:42.954 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:42.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:42Z" 2024-06-15T10:00:42.956 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:42.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:47.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:47.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:47.959 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:48.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:48.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:48.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:48.050+0000 7fbff3d3ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:48.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:48.050+0000 7fbff3d3ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:48.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:48.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:48.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:48.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:48.076 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:48.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:48.076 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:48Z" 2024-06-15T10:00:48.076 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:48.077 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:00:53.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:53.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:53.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:53.124 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:53.125 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:53.169 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:53.166+0000 7f21e28d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:53.171 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:53.170+0000 7f21e28d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:53.190 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:53.191 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:53.192 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:00:53.193 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:53.194 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:53Z" 2024-06-15T10:00:53.195 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:53.196 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:00:58.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:58.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:00:58.198 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:00:58.241 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:00:58.242 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:00:58.300 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:58.298+0000 7fd710d1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:58.301 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:00:58.298+0000 7fd710d1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:00:58.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:58.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:58.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:00:58.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:00:58.328 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:00:58.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:00:58.328 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:00:58Z" 2024-06-15T10:00:58.328 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:00:58.329 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:01:03.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:03.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:03.330 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:03.390 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:03.391 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:03.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:03.430+0000 7f3239830a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:03.431 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:03.430+0000 7f3239830a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:03.450 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:03.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:03.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:01:03.453 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:03.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:03Z" 2024-06-15T10:01:03.455 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:03.456 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:01:08.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:08.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:08.458 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:08.503 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:08.503 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:08.557 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:08.554+0000 7fb9fd85ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:08.558 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:08.554+0000 7fb9fd85ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:08.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:08.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.579 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:08.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:08.581 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:08Z" 2024-06-15T10:01:08.582 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:08.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:01:13.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:13.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:13.586 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:13.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:13.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:13.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:13.670+0000 7fc3e5f4ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:13.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:13.670+0000 7fc3e5f4ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:13.693 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:13.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:13.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:13.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:13.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:13.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:13Z" 2024-06-15T10:01:13.699 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:13.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:01:18.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:18.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:18.701 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:18.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:18.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:18.838 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:18.834+0000 7f9168c95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:18.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:18.838+0000 7f9168c95a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:18.859 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:18.859 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:18.859 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:18.859 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:18.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:18.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:01:18.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:18.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:18Z" 2024-06-15T10:01:18.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:18.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:01:23.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:23.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:23.870 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:23.915 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:23.915 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:23.969 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:23.970+0000 7f230e2d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:23.970 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:23.970+0000 7f230e2d4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:23.990 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:23.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:23.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:23.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.995 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:23.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:23.996 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:23.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:23.996 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:23Z" 2024-06-15T10:01:23.996 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:23.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:01:28.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-06-15T10:01:28.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:28.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:29.000 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:29.228 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:01:29.253 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:01:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:01:29.255 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:01:29.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:01:29.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:29.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:29.257 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:29.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:29.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:29.357 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:29.358+0000 7f0e7e108a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:29.358 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:29.358+0000 7f0e7e108a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:29.382 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:29.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:29.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:29.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:29.387 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:29Z" 2024-06-15T10:01:29.388 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:29.389 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:01:34.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:34.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:34.391 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:34.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:34.449 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:34.492 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:34.493+0000 7fc7cf30aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:34.492 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:34.493+0000 7fc7cf30aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:34.513 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:34.513 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:34.513 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:34.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:34.515 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:34.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:34.517 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:34Z" 2024-06-15T10:01:34.518 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:34.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:01:39.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:39.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:39.521 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:39.567 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:39.612 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:39.613+0000 7f7e3ee68a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:39.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:39.613+0000 7f7e3ee68a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:39.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:39.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:39.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:39.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:39.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:39.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:39.640 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:39Z" 2024-06-15T10:01:39.640 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:39.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:01:44.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:44.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:44.642 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:44.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:44.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:44.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:44.753+0000 7fb1a579ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:44.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:44.753+0000 7fb1a579ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:44.778 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:44.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:44.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:44.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:44.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:44Z" 2024-06-15T10:01:44.783 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:44.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:01:49.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:49.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:49.785 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:49.828 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:49.828 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:49.889 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:49.889+0000 7f06f7f29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:49.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:49.889+0000 7f06f7f29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:49.911 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:49.911 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:49.911 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:49.911 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:49.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:49.913 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:49.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:49.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:01:49.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:49.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:49.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:49.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:49.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:49.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:49.920 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:49Z" 2024-06-15T10:01:49.920 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:49.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:01:54.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:54.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:01:54.923 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:01:54.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:01:54.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:01:55.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:55.021+0000 7fd8f7f3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:55.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:01:55.021+0000 7fd8f7f3aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:01:55.043 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:01:55.043 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:01:55.043 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:01:55.043 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:55.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:55.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:55.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:01:55.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:01:55Z" 2024-06-15T10:01:55.048 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:01:55.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:02:00.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:00.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:00.051 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:00.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:00.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:00.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:00.137+0000 7fee4830ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:00.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:00.137+0000 7fee4830ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.156 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:00.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:00.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:00Z" 2024-06-15T10:02:00.161 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:00.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:05.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:05.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:05.164 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:05.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:05.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:05.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:05.261+0000 7fb203869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:05.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:05.261+0000 7fb203869a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:05.280 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:05.281 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.282 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:05.283 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:05Z" 2024-06-15T10:02:05.284 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:05.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:02:10.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:10.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:10.287 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:10.329 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:10.330 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:10.370 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:10.369+0000 7f9d5e11da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:10.370 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:10.373+0000 7f9d5e11da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:10.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:10.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:10.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:10.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:10Z" 2024-06-15T10:02:10.394 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:10.395 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:15.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:15.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:15.396 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:15.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:15.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:15.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:15.485+0000 7f891e0dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:15.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:15.485+0000 7f891e0dea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:15.502 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:15.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:15.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:15.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:15.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:15Z" 2024-06-15T10:02:15.507 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:15.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:02:20.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:20.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:20.510 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:20.555 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:20.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:20.612 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:20.613+0000 7f3bc1f09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:20.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:20.613+0000 7f3bc1f09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:20.648 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:20.648 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:20.648 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:20.648 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:20.648 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:20.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:20.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:20.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:20.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:20Z" 2024-06-15T10:02:20.653 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:20.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:02:25.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:25.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:25.656 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:25.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:25.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:25.750 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:25.748+0000 7f67be270a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:25.754 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:25.752+0000 7f67be270a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:25.775 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:25.775 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:25.775 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:25.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:25.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:25.778 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:25.779 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:25.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:25Z" 2024-06-15T10:02:25.781 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:25.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:30.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:30.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:30.783 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:30.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:30.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:30.880 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:30.880+0000 7f0d44be6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:30.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:30.880+0000 7f0d44be6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:30.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:30.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:30.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:30.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:30.908 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:30Z" 2024-06-15T10:02:30.909 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:30.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:35.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:35.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:35.911 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:35.954 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:35.955 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:36.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:36.012+0000 7fde0057ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:36.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:36.016+0000 7fde0057ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:36.041 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:36.041 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:36.041 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.042 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:36.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:36.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:36.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:36.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:36.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:36.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:36.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:36.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:36.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:36.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:36.049 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:36.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:36.049 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:36Z" 2024-06-15T10:02:36.049 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:36.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:41.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:41.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:41.051 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:41.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:41.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:41.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:41.136+0000 7f6ffa242a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:41.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:41.136+0000 7f6ffa242a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:41.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:41.157 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:41.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:41.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:41.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:41.161 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:41Z" 2024-06-15T10:02:41.162 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:41.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:46.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:46.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:46.165 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:46.207 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:46.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:46.251 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:46.252+0000 7ff3375f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:46.252 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:46.252+0000 7ff3375f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:46.270 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:46.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:46.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:46.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:46.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:46Z" 2024-06-15T10:02:46.275 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:46.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:51.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:51.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:51.277 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:51.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:51.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:51.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:51.368+0000 7f346dea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:51.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:51.372+0000 7f346dea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:51.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:51.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:51.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:51.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:51.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:51Z" 2024-06-15T10:02:51.397 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:51.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:02:56.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:56.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:02:56.400 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:02:56.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:02:56.443 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:02:56.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:56.488+0000 7f6fa55fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:56.490 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:02:56.488+0000 7f6fa55fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:02:56.508 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:02:56.508 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:02:56.508 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:02:56.508 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:02:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:56.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:56.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:02:56.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:02:56Z" 2024-06-15T10:02:56.513 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:02:56.514 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:03:01.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:01.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:01.515 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:01.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:01.559 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:01.610 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:01.608+0000 7fefb42dba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:01.611 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:01.612+0000 7fefb42dba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:01.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:01.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:01.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:01.638 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:01Z" 2024-06-15T10:03:01.639 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:01.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:03:06.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:06.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:06.642 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:06.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:06.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:06.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:06.732+0000 7f511a734a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:06.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:06.732+0000 7f511a734a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:06.772 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:06.772 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:06.772 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:06.772 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:06.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:06.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.769917_671.1", 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.769917Z", 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:06.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:06.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:06Z" 2024-06-15T10:03:06.777 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:06.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.769917_671.1 2024-06-15T10:03:11.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:11.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:11.779 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:11.823 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:11.823 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:11.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:11.864+0000 7fd6af9eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:11.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:11.864+0000 7fd6af9eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:11.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:11.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:11.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:11.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:11.888 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:11Z" 2024-06-15T10:03:11.888 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:11.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:03:16.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:16.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:16.890 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:16.933 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:16.933 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:16.980 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:16.979+0000 7fb68e234a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:16.980 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:16.979+0000 7fb68e234a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:16.999 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:17.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:17.001 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:17.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:17.003 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.004 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:17.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:17.005 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:17.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:17.005 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:16Z" 2024-06-15T10:03:17.005 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:17.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:03:22.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:22.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:22.007 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:22.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:22.053 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:22.113 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:22.111+0000 7f4e209f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:22.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:22.111+0000 7f4e209f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:22.142 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:22.144 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:22.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:22.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:22Z" 2024-06-15T10:03:22.148 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:22.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:03:27.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:27.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:27.153 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:27.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:27.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:27.238 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:27.235+0000 7fa9c95e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:27.239 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:27.239+0000 7fa9c95e2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.258 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:27.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:27.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:27.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:27Z" 2024-06-15T10:03:27.262 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:27.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.915314_676.1 2024-06-15T10:03:32.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:32.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:32.265 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:32.308 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:32.308 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:32.365 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:32.363+0000 7f0a1707ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:32.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:32.363+0000 7f0a1707ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:32.387 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:32.387 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:32.387 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:32.387 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:32.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:32.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:32.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:32.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:32.392 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:32Z" 2024-06-15T10:03:32.392 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:32.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:03:37.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:37.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:37.394 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:37.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:37.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:37.482 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:37.479+0000 7ff36798fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:37.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:37.483+0000 7ff36798fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:37.502 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:37.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:37.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:37.506 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:37Z" 2024-06-15T10:03:37.507 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:37.508 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:03:42.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:42.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:42.510 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:42.554 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:42.554 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:42.601 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:42.599+0000 7fe7e6c80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:42.602 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:42.599+0000 7fe7e6c80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:42.622 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:42.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:42.624 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.625 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:42.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.627 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:42.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:42.628 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:42.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:42.628 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:42Z" 2024-06-15T10:03:42.628 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:42.629 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:03:47.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:47.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:47.631 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:47.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:47.670 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:47.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:47.715+0000 7fcb0e9e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:47.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:47.715+0000 7fcb0e9e6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:47.738 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:47.738 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:47.739 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:47.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:03:47.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:47.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:47.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:47.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:47.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:47.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:47.744 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:47.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:47.744 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:47Z" 2024-06-15T10:03:47.744 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:47.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:03:52.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:52.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:52.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:52.871 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:52.871 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:52.923 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:52.923+0000 7ffa000b0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:52.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:52.923+0000 7ffa000b0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:52.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:52.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:52.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:52.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:52Z" 2024-06-15T10:03:52.952 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:52.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:03:57.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:57.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:03:57.955 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:03:57.999 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:03:58.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:03:58.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:58.043+0000 7f6e234f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:58.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:03:58.043+0000 7f6e234f8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:03:58.064 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:03:58.064 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:03:58.064 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:58.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:58.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:58.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:58.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:03:58Z" 2024-06-15T10:03:58.069 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:03:58.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:03.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:03.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:03.072 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:03.138 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:03.138 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:03.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:03.179+0000 7f45d1505a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:03.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:03.179+0000 7f45d1505a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:03.198 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:03.198 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:03.198 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:03.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:03.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:03.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:03.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:03Z" 2024-06-15T10:04:03.203 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:03.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:08.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:08.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:08.205 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:08.249 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:08.249 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:08.293 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:08.291+0000 7f16bf556a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:08.293 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:08.291+0000 7f16bf556a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:08.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:08.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:08.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:08.317 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:08Z" 2024-06-15T10:04:08.318 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:08.319 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:13.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:13.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:13.320 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:13.364 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:13.365 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:13.408 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:13.406+0000 7fa8335f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:13.409 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:13.406+0000 7fa8335f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.428 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.429 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:13.430 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:13.431 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:13.432 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:13.433 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:13.433 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:13Z" 2024-06-15T10:04:13.433 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:13.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:18.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:18.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:18.435 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:18.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:18.486 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:18.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:18.534+0000 7f3ce2fbca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:18.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:18.538+0000 7f3ce2fbca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:18.557 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:18.557 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:18.557 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:18.557 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:18.557 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:18.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:18.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.563 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:18.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:18.564 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:18.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:18.564 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:18Z" 2024-06-15T10:04:18.564 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:18.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:04:23.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:23.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:23.568 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:23.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:23.614 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:23.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:23.658+0000 7fb768015a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:23.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:23.662+0000 7fb768015a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:23.684 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:23.684 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:23.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:23.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:23.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.688 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:23.689 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:23.689 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:23.689 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:23.689 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:23Z" 2024-06-15T10:04:23.689 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:23.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:28.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:28.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:28.691 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:28.734 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:28.734 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:28.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:28.782+0000 7fb2ebb99a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:28.787 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:28.786+0000 7fb2ebb99a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:28.808 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:28Z" 2024-06-15T10:04:28.814 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:28.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:04:33.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:33.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:33.817 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:33.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:33.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:33.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:33.922+0000 7f1dfaa58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:33.926 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:33.922+0000 7f1dfaa58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:33.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:33.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:33.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:33.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:33.950 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:33Z" 2024-06-15T10:04:33.950 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:33.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:04:38.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:38.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:38.953 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:38.995 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:38.995 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:39.040 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:39.038+0000 7f3fec26da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:39.041 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:39.038+0000 7f3fec26da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:39.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:39.063 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:39.064 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:39.065 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:39.066 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:39.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:39.066 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:39Z" 2024-06-15T10:04:39.066 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:39.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:44.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:44.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:44.068 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:44.111 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:44.111 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:44.162 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:44.158+0000 7f7120e15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:44.163 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:44.162+0000 7f7120e15a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:44.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:44.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:44.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:44.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:44.190 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:44.190 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:44.190 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:44Z" 2024-06-15T10:04:44.190 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:44.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:04:49.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:49.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:49.192 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:49.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:49.234 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:49.293 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:49.290+0000 7f3027983a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:49.294 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:49.290+0000 7f3027983a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:49.312 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:49.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:49.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:04:49.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:49.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:49Z" 2024-06-15T10:04:49.317 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:49.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:04:54.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:54.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:54.319 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:54.364 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:54.364 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:54.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:54.414+0000 7f17b4ab9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:54.414 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:54.414+0000 7f17b4ab9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.439 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:54.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:54.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:54.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:54.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:54.444 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:54Z" 2024-06-15T10:04:54.444 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:54.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:04:59.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:59.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:04:59.446 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:04:59.488 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:04:59.488 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:04:59.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:59.546+0000 7f2255065a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:59.543 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:04:59.546+0000 7f2255065a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:04:59.571 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:04:59.571 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:04:59.571 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:04:59.571 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:04:59.572 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:59.573 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:59.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:04:59.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:04:59Z" 2024-06-15T10:04:59.576 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:04:59.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:05:04.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:04.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:04.579 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:04.621 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:04.621 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:04.669 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:04.669+0000 7f5130e9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:04.670 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:04.669+0000 7f5130e9ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:04.690 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:04.690 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:04.690 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:04.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:04.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:04.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:04.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:04Z" 2024-06-15T10:05:04.695 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:04.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:05:09.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:09.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:09.697 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:09.741 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:09.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:09.785+0000 7f899a147a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:09.785+0000 7f899a147a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:09.803 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:09.803 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:09.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:09.805 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:09.806 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:09.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:09Z" 2024-06-15T10:05:09.808 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:09.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:05:14.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:14.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:14.811 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:14.853 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:14.854 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:14.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:14.897+0000 7f5755e05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:14.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:14.897+0000 7f5755e05a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:14.915 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:05:14.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:14.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:14Z" 2024-06-15T10:05:14.920 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:14.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:05:19.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:19.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:19.923 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:19.964 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:19.964 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:20.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:20.017+0000 7f9d1476fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:20.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:20.017+0000 7f9d1476fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:20.034 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:20.034 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:20.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:20.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:20.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:20Z" 2024-06-15T10:05:20.040 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:20.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:05:25.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:25.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:25.042 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:25.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:25.087 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:25.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:25.141+0000 7fec37282a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:25.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:25.145+0000 7fec37282a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:25.162 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:25.162 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:25.162 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:25.162 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:25.162 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:25.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:25.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:25.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:25.166 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:25Z" 2024-06-15T10:05:25.167 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:25.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:05:30.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:30.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:30.169 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:30.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:30.214 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:30.270 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:30.269+0000 7f7fad172a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:30.271 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:30.273+0000 7f7fad172a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:30.291 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:30.291 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:30.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:30.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:30.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:30.295 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:30.296 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:30Z" 2024-06-15T10:05:30.297 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:30.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:05:35.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:35.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:35.299 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:35.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:35.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:35.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:35.381+0000 7f7c37b64a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:35.380 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:35.381+0000 7f7c37b64a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:35.401 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:35.402 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:35.403 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:35.404 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:35.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:35Z" 2024-06-15T10:05:35.406 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:35.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:05:40.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:40.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:40.408 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:40.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:40.452 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:40.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:40.501+0000 7f4e19478a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:40.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:40.501+0000 7f4e19478a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:40.522 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:40.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:40.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:40.526 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:40.527 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:40.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:40.528 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:40Z" 2024-06-15T10:05:40.528 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:40.529 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:05:45.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:45.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:45.530 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:45.573 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:45.573 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:45.631 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:45.633+0000 7fdfe03cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:45.632 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:45.633+0000 7fdfe03cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.655 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:45.656 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:45.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:45.658 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:45Z" 2024-06-15T10:05:45.659 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:45.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:05:50.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:50.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:50.662 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:50.711 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:50.711 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:50.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:50.773+0000 7fa7097b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:50.774 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:50.773+0000 7fa7097b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:50.793 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:50.793 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:50.794 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:50.795 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:05:50.796 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:50.797 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:50Z" 2024-06-15T10:05:50.798 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:50.799 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:05:55.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:55.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:05:55.800 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:05:55.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:05:55.843 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:05:55.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:55.892+0000 7f13034d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:55.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:05:55.896+0000 7f13034d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:05:55.914 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:05:55.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:05:55.914 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:05:55.914 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:05:55.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:05:55.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:55.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:55.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:05:55.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:05:55Z" 2024-06-15T10:05:55.921 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:05:55.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:00.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:00.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:00.924 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:01.058 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:01.059 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:01.108 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:01.108+0000 7fc8a484aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:01.109 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:01.108+0000 7fc8a484aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:01.128 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:01.128 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:01.128 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:01.128 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:01.128 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:01.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:01.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:01.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.132 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:01.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:01.133 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:01.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:01.133 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:01Z" 2024-06-15T10:06:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:01.134 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:06.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:06.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:06.138 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:06.199 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:06.199 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:06.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:06.260+0000 7f08182d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:06.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:06.260+0000 7f08182d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.286 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:06.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:06:06.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:06.289 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:06Z" 2024-06-15T10:06:06.290 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:06.291 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:06:11.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:11.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:11.292 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:11.335 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:11.335 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:11.387 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:11.388+0000 7f5d19d9da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:11.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:11.388+0000 7f5d19d9da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:11.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:11.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:11.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:11.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:11.410 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:11Z" 2024-06-15T10:06:11.411 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:11.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:06:16.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:16.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:16.413 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:16.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:16.457 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:16.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:16.508+0000 7fab131f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:16.510 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:16.508+0000 7fab131f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:16.528 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:16.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:16.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:16.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:16.533 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:16.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:16.534 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:16Z" 2024-06-15T10:06:16.534 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:16.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:21.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:21.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:21.536 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:21.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:21.675 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:21.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:21.716+0000 7f3369afda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:21.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:21.720+0000 7f3369afda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:21.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:21.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:21.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:21.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:21.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:21.743 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:21Z" 2024-06-15T10:06:21.744 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:21.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:26.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:26.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:26.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:26.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:26.790 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:26.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:26.840+0000 7fcdcb5b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:26.840 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:26.840+0000 7fcdcb5b8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:26.859 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:26.860 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:26.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:26.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:26.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:26.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:26Z" 2024-06-15T10:06:26.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:26.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:06:31.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:31.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:31.865 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:31.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:31.907 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:31.960 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:31.960+0000 7fc39895ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:31.961 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:31.960+0000 7fc39895ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:31.981 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:31.981 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:31.981 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:31.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:31.983 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:31.984 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:31.985 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:31Z" 2024-06-15T10:06:31.986 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:31.987 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:06:36.989 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... FAIL 2024-06-15T10:06:36.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:36.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:36.990 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:37.033 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:37.085 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:06:37.107 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:06:37.108 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:37.108 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:06:37.108 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:06:37.108 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:06:37.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:06:37.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:37.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:37.109 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:37.152 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:37.152 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:37.202 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:37.200+0000 7f093a5a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:37.203 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:37.204+0000 7f093a5a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:37.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:37.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:37.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:37.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:37.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:37.228 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:37Z" 2024-06-15T10:06:37.229 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:37.230 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:42.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:42.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:42.232 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:42.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:42.275 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:42.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:42.340+0000 7f7901456a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:42.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:42.340+0000 7f7901456a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:42.367 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:42.367 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:42.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:42.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:42.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:42.371 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:42Z" 2024-06-15T10:06:42.372 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:42.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:47.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:47.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:47.375 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:47.422 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:47.426 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:47.465 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:47.463+0000 7f4358f5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:47.466 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:47.463+0000 7f4358f5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.486 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:47.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.488 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:47.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:47.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:47Z" 2024-06-15T10:06:47.491 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:47.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:06:52.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:52.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:52.493 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:52.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:52.538 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:52.590 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:52.587+0000 7ff75ca6ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:52.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:52.591+0000 7ff75ca6ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:52.617 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:52.617 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:52.618 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:52.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:52.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:52.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:52Z" 2024-06-15T10:06:52.622 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:52.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:06:57.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:57.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:06:57.624 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:06:57.669 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:06:57.669 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:06:57.730 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:57.727+0000 7f0c2a575a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:57.731 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:06:57.731+0000 7f0c2a575a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:06:57.758 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:06:57.759 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:06:57.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:06:57.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:06:57.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:06:57Z" 2024-06-15T10:06:57.763 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:06:57.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:02.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:02.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:02.765 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:02.808 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:02.808 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:02.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:02.859+0000 7f474241ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:02.863 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:02.863+0000 7f474241ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:02.888 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:02.889 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:02.890 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:02.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:02.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:02Z" 2024-06-15T10:07:02.893 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:02.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:07.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:07.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:07.896 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:07.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:07.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:07.984 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:07.983+0000 7f63e3302a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:07.985 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:07.983+0000 7f63e3302a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:08.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:08.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:08.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:07:08.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:08.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:08.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:08.017 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:07Z" 2024-06-15T10:07:08.017 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:08.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:07:13.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:13.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:13.019 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:13.066 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:13.066 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:13.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:13.135+0000 7fe7b8378a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:13.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:13.135+0000 7fe7b8378a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.157 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:13.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:13.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:13.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:13.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:13.162 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:13.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:13.163 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:13Z" 2024-06-15T10:07:13.163 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:13.164 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:18.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:18.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:18.165 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:18.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:18.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:18.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:18.247+0000 7fc8e737da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:18.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:18.247+0000 7fc8e737da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.269 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:18.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:18.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:18Z" 2024-06-15T10:07:18.273 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:18.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:07:23.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:23.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:23.276 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:23.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:23.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:23.374 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:23.371+0000 7f341822ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:23.374 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:23.375+0000 7f341822ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:23.394 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:23.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:23.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:23.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:23.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:23Z" 2024-06-15T10:07:23.399 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:23.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:28.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:28.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:28.402 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:28.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:28.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:28.491 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:28.491+0000 7f87b3e59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:28.492 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:28.491+0000 7f87b3e59a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:28.512 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:28.512 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:28.512 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:28.512 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:28.512 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:28.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:28.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:28.515 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:28.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:28Z" 2024-06-15T10:07:28.517 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:28.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:33.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:33.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:33.520 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:33.564 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:33.564 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:33.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:33.615+0000 7f1c7bfada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:33.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:33.619+0000 7f1c7bfada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:33.637 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:33.638 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:33.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:33.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:33.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:33Z" 2024-06-15T10:07:33.642 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:33.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:38.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:38.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:38.645 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:38.688 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:38.688 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:38.779 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:38.778+0000 7f17cc8b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:38.780 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:38.778+0000 7f17cc8b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:38.806 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.807 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:38.808 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:38.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.810 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:38.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:38.811 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:38.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:38.811 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:38Z" 2024-06-15T10:07:38.811 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:38.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:07:43.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:43.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:43.813 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:43.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:43.857 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:43.901 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:43.898+0000 7fdc7a5d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:43.902 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:43.898+0000 7fdc7a5d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:43.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.923 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.924 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:43.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.927 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:43.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:43.927 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:43.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:43.928 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:43Z" 2024-06-15T10:07:43.928 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:43.929 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:07:48.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:48.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:48.930 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:48.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:48.974 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:49.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:49.014+0000 7f3974ceda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:49.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:49.014+0000 7f3974ceda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:49.036 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:49.036 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:49.036 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:49.036 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:49.036 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:49.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:49.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:49.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:49.040 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:49Z" 2024-06-15T10:07:49.041 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:49.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:07:54.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:54.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:54.044 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:54.088 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:54.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:54.130 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:54.126+0000 7f335833ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:54.131 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:54.130+0000 7f335833ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:54.157 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:54.157 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:54.157 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:54.157 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:54.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:54.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:54.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:54.161 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:54Z" 2024-06-15T10:07:54.162 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:54.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:07:59.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:59.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:07:59.164 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:07:59.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:07:59.208 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:07:59.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:59.246+0000 7f5811227a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:59.251 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:07:59.250+0000 7f5811227a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:07:59.269 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:07:59.269 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:59.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:59.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:59.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:07:59.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:07:59Z" 2024-06-15T10:07:59.274 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:07:59.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:08:04.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:04.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:04.276 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:04.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:04.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:04.383 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:04.382+0000 7effde75ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:04.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:04.382+0000 7effde75ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:04.407 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:04.407 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:04.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:04.409 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:08:04.410 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:04.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:04Z" 2024-06-15T10:08:04.412 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:04.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:08:09.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:09.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:09.414 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:09.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:09.458 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:09.533 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:09.530+0000 7fee4c13da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:09.534 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:09.530+0000 7fee4c13da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:09.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:09.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:09.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.566 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:09.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:09.567 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:09.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:09.567 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:09Z" 2024-06-15T10:08:09.567 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:09.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:08:14.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:14.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:14.569 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:14.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:14.612 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:14.658 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:14.654+0000 7f84b145ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:14.659 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:14.658+0000 7f84b145ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:14.693 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:14.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:14.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:14.697 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:14.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:14Z" 2024-06-15T10:08:14.699 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:14.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:08:19.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:19.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:19.701 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:19.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:19.744 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:19.803 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:19.806+0000 7f5fddaa5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:19.804 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:19.806+0000 7f5fddaa5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:19.827 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:19.827 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:19.827 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:19.827 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:19.827 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:19.828 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:19.829 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:19.830 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:19.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:19Z" 2024-06-15T10:08:19.832 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:19.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:08:24.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:24.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:24.834 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:24.878 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:24.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:24.939 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:24.942+0000 7f4f4c346a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:24.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:24.942+0000 7f4f4c346a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:24.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:24.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:24.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:24.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:24.963 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:24Z" 2024-06-15T10:08:24.964 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:24.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:08:29.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:29.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:29.966 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:30.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:30.026 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:30.085 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:30.085+0000 7f20c9a7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:30.086 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:30.085+0000 7f20c9a7fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:30.111 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:30.111 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:30.111 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:30.111 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:30.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:30.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:08:30.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:30.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:30Z" 2024-06-15T10:08:30.116 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:30.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:08:35.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:35.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:35.118 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:35.162 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:35.162 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:35.241 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:35.241+0000 7f54296caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:35.241 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:35.241+0000 7f54296caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:35.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:35.259 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:35.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:35.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:35.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:35.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:35Z" 2024-06-15T10:08:35.264 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:35.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:08:40.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:40.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:40.267 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:40.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:40.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:40.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:40.373+0000 7f4b7ed29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:40.373 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:40.373+0000 7f4b7ed29a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:40.394 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:40.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:40.394 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:40.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:40.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:40.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:40.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:40.399 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:40Z" 2024-06-15T10:08:40.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:40.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:08:45.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:45.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:45.402 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:45.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:45.445 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:45.493 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:45.493+0000 7f37963b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:45.494 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:45.493+0000 7f37963b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:45.515 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:45.516 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.517 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:45.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:45Z" 2024-06-15T10:08:45.520 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:45.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:08:50.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:50.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:50.522 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:50.566 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:50.566 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:50.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:50.621+0000 7f4ad1b75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:50.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:50.621+0000 7f4ad1b75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:50.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:50.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:50.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:50.655 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:50Z" 2024-06-15T10:08:50.656 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:50.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:08:55.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:55.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:08:55.658 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:08:55.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:08:55.709 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:08:55.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:55.761+0000 7feabcd97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:55.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:08:55.761+0000 7feabcd97a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:55.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.780 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:55.781 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:08:55.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:08:55.783 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:08:55Z" 2024-06-15T10:08:55.784 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:08:55.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:09:00.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:00.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:00.786 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:00.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:00.830 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:00.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:00.893+0000 7f407c45da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:00.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:00.893+0000 7f407c45da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:00.915 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:00.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:00.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:00.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440137.190396_920.1", 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:57.190396Z", 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:00.919 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:00.920 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:00.921 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:00Z" 2024-06-15T10:09:00.921 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:00.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440137.190396_920.1 2024-06-15T10:09:05.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:05.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:05.923 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:05.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:05.967 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:06.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:06.021+0000 7f3ecb477a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:06.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:06.021+0000 7f3ecb477a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:06.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.047 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:06.048 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:06Z" 2024-06-15T10:09:06.048 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:06.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:09:11.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:11.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:11.051 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:11.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:11.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:11.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:11.137+0000 7f1faff7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:11.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:11.137+0000 7f1faff7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.156 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:11.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:11.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:11.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:11.160 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:11Z" 2024-06-15T10:09:11.161 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:11.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:09:16.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:16.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:16.163 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:16.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:16.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:16.255 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:16.257+0000 7f755d74da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:16.256 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:16.257+0000 7f755d74da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.275 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:16.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:16.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:16.278 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:16.279 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:16Z" 2024-06-15T10:09:16.280 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:16.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:09:21.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:21.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:21.282 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:21.412 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:21.412 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:21.477 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:21.476+0000 7fc9b19fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:21.478 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:21.476+0000 7fc9b19fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:21.503 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.504 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:21.505 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:21.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:21.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:21.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:21.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:21Z" 2024-06-15T10:09:21.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:21.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:09:26.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:26.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:26.512 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:26.561 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:26.561 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:26.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:26.608+0000 7f408afcfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:26.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:26.608+0000 7f408afcfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:26.633 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:26.633 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:26.633 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:26.633 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:26.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:26.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:26.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:26.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:26Z" 2024-06-15T10:09:26.638 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:26.639 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:09:31.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:31.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:31.641 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:31.682 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:31.682 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:31.732 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:31.732+0000 7f77a6109a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:31.733 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:31.732+0000 7f77a6109a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:31.759 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:31.760 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.761 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:31.762 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:31.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:31.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:31.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:31.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:31.767 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:31Z" 2024-06-15T10:09:31.767 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:31.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:09:36.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:36.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:36.769 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:36.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:36.816 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:36.874 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:36.872+0000 7fa1128ffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:36.876 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:36.876+0000 7fa1128ffa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:36.900 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:36.900 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:36.900 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:36.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:36.902 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:09:36.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:36.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:36Z" 2024-06-15T10:09:36.905 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:36.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:09:41.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:41.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:41.908 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:41.952 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:41.952 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:42.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:42.004+0000 7ff229a52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:42.005 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:42.004+0000 7ff229a52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:42.030 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:42.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:42.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:42.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:42Z" 2024-06-15T10:09:42.035 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:42.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:09:47.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:47.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:47.037 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:47.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:47.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:47.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:47.132+0000 7ffadc99ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:47.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:47.132+0000 7ffadc99ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:47.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:47.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:47.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:47.165 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:47Z" 2024-06-15T10:09:47.165 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:47.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:09:52.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:52.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:52.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:52.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:52.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:52.270 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:52.268+0000 7fc4a9a0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:52.271 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:52.272+0000 7fc4a9a0aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:52.290 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:52.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:52.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:52.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:52.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:52Z" 2024-06-15T10:09:52.295 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:52.296 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:09:57.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:57.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:09:57.298 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:09:57.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:09:57.340 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:09:57.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:57.388+0000 7ff9105cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:57.390 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:09:57.388+0000 7ff9105cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:09:57.415 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:57.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:57.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:09:57.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:09:57.419 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:09:57Z" 2024-06-15T10:09:57.420 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:09:57.420 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:02.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:02.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:02.422 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:02.467 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:02.467 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:02.528 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:02.528+0000 7f452bf52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:02.529 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:02.528+0000 7f452bf52a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.554 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:02.555 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:02.557 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:02.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.560 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:02.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:02.561 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:02.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:02.562 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:02Z" 2024-06-15T10:10:02.562 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:02.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:07.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:07.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:07.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:07.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:07.611 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:07.669 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:07.668+0000 7f624cbb5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:07.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:07.672+0000 7f624cbb5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:07.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:07.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:07.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:07.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:07.697 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:07.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:07.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:07.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:07.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:07Z" 2024-06-15T10:10:07.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:07.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:10:12.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:12.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:12.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:12.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:12.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:12.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:12.811+0000 7f35e19b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:12.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:12.811+0000 7f35e19b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:12.832 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:12.832 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:12.832 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:12.832 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:12.832 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:12.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:12.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.836 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:12.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:12.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:12.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:12.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:12Z" 2024-06-15T10:10:12.837 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:12.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:10:17.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:17.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:17.840 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:17.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:17.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:17.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:17.923+0000 7fcc25fc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:17.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:17.923+0000 7fcc25fc5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:17.949 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:17.949 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:17.949 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:17.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:17.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:17.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:17.954 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.955 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:17.957 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:17Z" 2024-06-15T10:10:17.957 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:17.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:10:22.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:22.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:22.960 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:23.003 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:23.004 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:23.054 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:23.051+0000 7fd76b526a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:23.055 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:23.055+0000 7fd76b526a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:23.074 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:23.074 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:23.074 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:23.074 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:23.074 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:23.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:23.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:23.077 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:23.078 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:23Z" 2024-06-15T10:10:23.079 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:23.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:28.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:28.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:28.082 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:28.128 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:28.128 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:28.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:28.179+0000 7f99df3bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:28.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:28.179+0000 7f99df3bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:28.198 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:28.198 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:28.198 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:28.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:28.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:10:28.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:28.202 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:28Z" 2024-06-15T10:10:28.203 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:28.204 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:33.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:33.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:33.206 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:33.250 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:33.251 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:33.302 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:33.299+0000 7f9b9e17ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:33.303 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:33.303+0000 7f9b9e17ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.323 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:33.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:33.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:10:33.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:33.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:33Z" 2024-06-15T10:10:33.328 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:33.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:38.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:38.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:38.330 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:38.374 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:38.375 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:38.419 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:38.419+0000 7f895ea5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:38.420 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:38.419+0000 7f895ea5da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:38.440 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:38.440 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:38.440 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:38.440 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:38.440 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:38.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:38.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:38.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:38.444 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:38Z" 2024-06-15T10:10:38.445 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:38.446 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:10:43.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:43.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:43.447 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:43.490 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:43.491 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:43.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:43.535+0000 7ff215e11a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:43.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:43.535+0000 7ff215e11a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:43.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440137.061971_914.1", 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:57.061971Z", 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:43.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:43.562 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:43.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:43.563 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:43Z" 2024-06-15T10:10:43.563 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:43.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440137.061971_914.1 2024-06-15T10:10:48.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:48.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:48.565 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:48.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:48.609 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:48.661 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:48.659+0000 7f7d4f08ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:48.662 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:48.659+0000 7f7d4f08ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:48.682 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:48.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:48.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:48.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:10:48.687 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:48.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:48.689 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:48.689 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:10:48.689 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:48.690 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:48.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:48.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:48.693 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:48.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:48.693 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:48Z" 2024-06-15T10:10:48.693 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:48.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:10:53.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:53.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:53.697 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:53.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:53.742 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:53.806 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:53.803+0000 7fb9eb3f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:53.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:53.807+0000 7fb9eb3f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:53.828 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.829 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:53.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:53.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:53.833 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:53.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:53.834 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:53Z" 2024-06-15T10:10:53.834 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:53.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:10:58.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:58.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:10:58.836 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:10:58.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:10:58.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:10:58.923 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:58.923+0000 7f8b6d488a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:58.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:10:58.923+0000 7f8b6d488a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:10:58.943 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:10:58.943 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:58.944 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:58.945 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:58.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:10:58.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:10:58Z" 2024-06-15T10:10:58.948 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:10:58.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:11:03.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:03.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:03.951 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:03.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:03.995 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:04.051 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:04.051+0000 7fc60bf81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:04.052 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:04.051+0000 7fc60bf81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:04.073 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:04.073 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:04.073 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:04.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:04.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.760139_669.1", 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:04.076 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.760139Z", 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:04.077 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:04Z" 2024-06-15T10:11:04.078 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:04.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.760139_669.1 2024-06-15T10:11:09.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:09.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:09.082 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:09.128 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:09.128 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:09.190 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:09.186+0000 7ff31ef0ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:09.191 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:09.190+0000 7ff31ef0ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:09.215 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:09.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:09.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:09.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:09.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:09Z" 2024-06-15T10:11:09.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:09.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:11:14.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:14.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:14.223 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:14.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:14.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:14.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:14.318+0000 7f3a253b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:14.323 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:14.322+0000 7f3a253b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:14.346 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:14.347 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:14.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:11:14.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:14.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:14.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:14Z" 2024-06-15T10:11:14.352 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:14.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:11:19.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:19.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:19.355 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:19.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:19.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:19.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:19.438+0000 7fae17662a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:19.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:19.438+0000 7fae17662a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:19.460 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:19.460 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:19.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:19.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:19.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:19.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:19Z" 2024-06-15T10:11:19.465 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:19.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:11:24.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:24.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:24.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:24.512 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:24.512 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:24.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:24.558+0000 7f88bc7c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:24.567 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:24.562+0000 7f88bc7c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:24.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:24.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:24.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:24.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:24.588 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:24.589 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:24.589 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:24Z" 2024-06-15T10:11:24.589 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:24.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:11:29.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:29.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:29.591 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:29.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:29.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:29.689 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:29.686+0000 7f6b02d0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:29.690 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:29.686+0000 7f6b02d0fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:29.717 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:29.717 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:29.717 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.718 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:29.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:29.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:29.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:29.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:29.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:11:29.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:29.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:29.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:29.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:29.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:29.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:29.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:29.727 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:29.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:29.727 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:29Z" 2024-06-15T10:11:29.727 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:29.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:11:34.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:34.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:34.731 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:34.774 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:34.774 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:34.821 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:34.818+0000 7f35406ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:34.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:34.818+0000 7f35406ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:34.840 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:34.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:34.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:34.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:34.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:34Z" 2024-06-15T10:11:34.845 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:34.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:11:39.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:39.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:39.848 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:39.891 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:39.892 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:39.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:39.934+0000 7fb6a86f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:39.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:39.934+0000 7fb6a86f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:39.956 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:39.957 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:39.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:39.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:11:39.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:39.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:39Z" 2024-06-15T10:11:39.962 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:39.963 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:11:44.965 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-06-15T10:11:44.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:44.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:44.966 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:45.010 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:45.010 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:45.060 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:11:45.087 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:11:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:11:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:11:45.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:11:45.088 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:11:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:11:45.089 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:11:45.089 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:11:45.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:11:45.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:45.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:45.090 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:45.133 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:45.134 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:45.195 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:45.194+0000 7fe5fba2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:45.196 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:45.194+0000 7fe5fba2ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:45.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:45.215 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:45.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:45.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:45.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:45.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:45Z" 2024-06-15T10:11:45.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:45.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:11:50.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:50.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:50.223 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:50.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:50.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:50.307 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:50.310+0000 7f8e64ffca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:50.308 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:50.310+0000 7f8e64ffca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:50.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:50.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:50.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:50.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.330 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:50.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:50.331 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:50.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:50.331 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:50Z" 2024-06-15T10:11:50.331 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:50.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:11:55.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:55.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:11:55.334 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:11:55.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:11:55.376 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:11:55.423 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:55.426+0000 7f9b596caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:55.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:11:55.426+0000 7f9b596caa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:11:55.447 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:55.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:11:55.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:55.451 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:11:55.452 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:11:55.453 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:55.454 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:11:55Z" 2024-06-15T10:11:55.455 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:11:55.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:12:00.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:00.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:00.459 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:00.503 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:00.503 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:00.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:00.565+0000 7fa789d8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:00.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:00.565+0000 7fa789d8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:00.583 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:00.583 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:00.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:00.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:00.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:00.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:00.588 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:00Z" 2024-06-15T10:12:00.589 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:00.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:12:05.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:05.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:05.591 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:05.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:05.635 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:05.684 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:05.685+0000 7fb5e5ba3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:05.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:05.685+0000 7fb5e5ba3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:05.711 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:05.711 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:05.711 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:05.711 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:05.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.760139_669.1", 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:05.714 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.760139Z", 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:05.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:05.716 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:05.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:05.717 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:05Z" 2024-06-15T10:12:05.717 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:05.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.760139_669.1 2024-06-15T10:12:10.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:10.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:10.720 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:10.763 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:10.763 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:10.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:10.821+0000 7f0aa4567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:10.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:10.821+0000 7f0aa4567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:10.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:10.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:10.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:10.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:10Z" 2024-06-15T10:12:10.849 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:10.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:12:15.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:15.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:15.851 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:15.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:15.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:15.942 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:15.941+0000 7f1786bdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:15.943 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:15.945+0000 7f1786bdaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:15.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:15.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:15.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:15.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:15.966 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:15Z" 2024-06-15T10:12:15.967 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:15.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:12:20.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:20.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:20.969 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:21.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:21.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:21.080 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:21.081+0000 7f54f0c3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:21.082 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:21.081+0000 7f54f0c3ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:21.118 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:21.118 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:21.119 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:21.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:21.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:21.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:21Z" 2024-06-15T10:12:21.123 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:21.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:12:26.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:26.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:26.125 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:26.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:26.168 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:26.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:26.209+0000 7fe313791a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:26.210 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:26.209+0000 7fe313791a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:26.228 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:26.228 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:26.229 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:26.230 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:26.231 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:26.232 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:26Z" 2024-06-15T10:12:26.233 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:26.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:12:31.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:31.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:31.235 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:31.277 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:31.277 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:31.326 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:31.325+0000 7fc7307a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:31.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:31.329+0000 7fc7307a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:31.351 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:31.351 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:31.351 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:31.351 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:31.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:31.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:31.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:31Z" 2024-06-15T10:12:31.356 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:31.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:12:36.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:36.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:36.358 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:36.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:36.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:36.457 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:36.457+0000 7f2419f92a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:36.458 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:36.461+0000 7f2419f92a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:36.484 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:36.485 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:36.486 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:36.487 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:36.488 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:36Z" 2024-06-15T10:12:36.489 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:36.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:12:41.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:41.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:41.492 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:41.534 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:41.535 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:41.590 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:41.589+0000 7f8964fcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:41.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:41.593+0000 7f8964fcda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.610 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.611 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:41.612 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:41.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:41.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:41.615 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:41Z" 2024-06-15T10:12:41.615 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:41.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:12:46.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:46.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:46.617 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:46.659 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:46.660 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:46.702 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:46.705+0000 7f0c16643a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:46.703 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:46.705+0000 7f0c16643a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:46.723 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:46.723 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:46.723 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:46.723 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:46.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:46.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:12:46.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:46.727 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:46Z" 2024-06-15T10:12:46.728 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:46.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:12:51.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:51.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:51.734 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:51.777 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:51.778 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:51.838 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:51.840+0000 7fc7285cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:51.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:51.840+0000 7fc7285cea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:51.859 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:51.860 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:51.861 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:51.862 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:51.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:51Z" 2024-06-15T10:12:51.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:51.865 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:12:56.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:56.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:12:56.867 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:12:56.912 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:12:56.912 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:12:56.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:56.976+0000 7f5499411a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:56.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:12:56.976+0000 7f5499411a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:12:57.000 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:12:57.000 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:57.001 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:57.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:57.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:12:57.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:12:57.005 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:12:56Z" 2024-06-15T10:12:57.006 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:12:57.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:13:02.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:02.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:02.008 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:02.052 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:02.052 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:02.105 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:02.104+0000 7f4e60cdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:02.106 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:02.108+0000 7f4e60cdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:02.127 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:02.127 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:02.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:02.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:02.130 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:02.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:02Z" 2024-06-15T10:13:02.132 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:02.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:13:07.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:07.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:07.135 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:07.179 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:07.179 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:07.241 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:07.240+0000 7fe5aeef0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:07.242 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:07.244+0000 7fe5aeef0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:07.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:07.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:07.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:07.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:07.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:07.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:07.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:07.275 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:07Z" 2024-06-15T10:13:07.275 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:07.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:13:12.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:12.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:12.277 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:12.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:12.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:12.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:12.360+0000 7f745e682a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:12.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:12.360+0000 7f745e682a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:12.383 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:12.383 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:12.383 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:12.383 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:12.383 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:12.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:12.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:12.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:12.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.388 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:12.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:12.389 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:12.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:12.389 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:12Z" 2024-06-15T10:13:12.389 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:12.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:13:17.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:17.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:17.392 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:17.436 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:17.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:17.487 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:17.488+0000 7f039a56aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:17.488 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:17.488+0000 7f039a56aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:17.507 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:17.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:17.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:17.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:17Z" 2024-06-15T10:13:17.512 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:17.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:13:22.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:22.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:22.515 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:22.560 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:22.561 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:22.613 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:22.612+0000 7f4bb3b37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:22.636 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:22.612+0000 7f4bb3b37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:22.637 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:22.637 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:22.637 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.638 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:22.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:22.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:22.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:22.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:22.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:22.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:22.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:22.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:22.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:22.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:22.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:22.644 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:22.645 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440136.709769_896.1", 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:56.709769Z", 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:22.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:22.648 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:22.649 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:22Z" 2024-06-15T10:13:22.650 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:22.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440136.709769_896.1 2024-06-15T10:13:27.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:27.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:27.652 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:27.698 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:27.698 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:27.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:27.788+0000 7fc1dfe02a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:27.790 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:27.792+0000 7fc1dfe02a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:27.815 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:27.816 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:27.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:27.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:27.819 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:27Z" 2024-06-15T10:13:27.820 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:27.821 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:13:32.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:32.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:32.822 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:32.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:32.865 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:32.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:32.908+0000 7f9c1f122a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:32.907 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:32.908+0000 7f9c1f122a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:32.926 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:32.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:32.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:32.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:32.930 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:32.931 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:32.932 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:32Z" 2024-06-15T10:13:32.932 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:32.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:13:37.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:37.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:37.935 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:37.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:37.978 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:38.019 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:38.020+0000 7f39fc901a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:38.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:38.020+0000 7f39fc901a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:38.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:38.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:38.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:38.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:38.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:38.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:38.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:38.044 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:38Z" 2024-06-15T10:13:38.044 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:38.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:13:43.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:43.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:43.047 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:43.095 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:43.096 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:43.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:43.135+0000 7f0990497a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:43.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:43.135+0000 7f0990497a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:43.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:43.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:43.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:43Z" 2024-06-15T10:13:43.164 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:43.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:13:48.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:48.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:48.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:48.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:48.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:48.264 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:48.263+0000 7fe89fc36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:48.265 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:48.263+0000 7fe89fc36a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:48.290 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:48.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:48.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:13:48.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:48.294 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:48.295 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:48.296 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:48Z" 2024-06-15T10:13:48.296 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:48.297 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:13:53.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:53.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:53.299 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:53.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:53.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:53.384 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:53.383+0000 7fb014411a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:53.385 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:53.383+0000 7fb014411a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:53.404 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:53.404 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:53.405 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:53.406 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:53.407 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:53.408 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:53Z" 2024-06-15T10:13:53.409 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:53.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:13:58.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:58.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:13:58.412 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:13:58.455 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:13:58.455 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:13:58.497 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:58.495+0000 7fcd3417aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:58.498 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:13:58.499+0000 7fcd3417aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:13:58.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:13:58.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:13:58.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:58.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:13:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:13:58.522 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:13:58.523 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:13:58Z" 2024-06-15T10:13:58.523 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:13:58.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:03.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:03.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:03.525 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:03.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:03.569 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:03.617 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:03.615+0000 7fb6347f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:03.618 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:03.619+0000 7fb6347f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:03.638 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:03.639 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:03.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:03.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:03.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:03.643 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:03.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:03.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:03Z" 2024-06-15T10:14:03.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:03.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:08.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:08.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:08.647 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:08.690 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:08.690 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:08.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:08.743+0000 7f4f2da8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:08.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:08.747+0000 7f4f2da8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:08.769 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:08.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:08.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:08.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:08.773 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:08Z" 2024-06-15T10:14:08.774 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:08.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:13.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:13.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:13.776 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:13.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:13.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:13.864 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:13.863+0000 7fcb2ad00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:13.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:13.863+0000 7fcb2ad00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.884 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:13.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:13.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:13.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:13.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:13.889 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:13.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:13.890 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:13Z" 2024-06-15T10:14:13.890 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:13.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:18.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:18.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:18.895 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:18.938 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:18.938 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:18.987 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:18.987+0000 7f38577b6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:18.988 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:18.987+0000 7f38577b6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:19.013 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:19.013 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:19.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:19.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:19.016 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:19.017 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:18Z" 2024-06-15T10:14:19.018 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:19.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:24.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:24.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:24.020 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:24.069 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:24.070 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:24.110 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:24.107+0000 7fae299a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:24.111 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:24.111+0000 7fae299a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:24.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:24.130 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.131 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:24.132 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:24.133 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:24.134 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:24.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:24.134 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:24Z" 2024-06-15T10:14:24.134 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:24.135 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:29.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:29.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:29.137 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:29.180 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:29.180 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:29.231 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:29.231+0000 7f9911043a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:29.232 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:29.231+0000 7f9911043a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:29.256 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:29.256 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:29.256 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:29.256 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:29.257 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:29.257 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:29.257 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:29.257 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:29.258 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:29.258 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:29.259 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:29.260 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:29.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:29Z" 2024-06-15T10:14:29.262 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:29.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:14:34.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:34.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:34.264 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:34.307 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:34.307 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:34.358 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:34.354+0000 7f16bd7e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:34.358+0000 7f16bd7e8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:34.383 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:34.383 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:34.383 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:34.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:34.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:34.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:34.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:34Z" 2024-06-15T10:14:34.388 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:34.389 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:14:39.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:39.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:39.392 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:39.435 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:39.435 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:39.476 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:39.474+0000 7f706831ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:39.477 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:39.474+0000 7f706831ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:39.495 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:39.496 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:39.497 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:39.498 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:39.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:39.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:39.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:39.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:39.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:14:39.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:39.505 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:39.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:39Z" 2024-06-15T10:14:39.507 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:39.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:14:44.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:44.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:44.510 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:44.557 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:44.557 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:44.615 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:44.614+0000 7f30da9f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:44.616 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:44.614+0000 7f30da9f3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:44.639 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.640 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:44.641 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:44.642 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:44.643 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:44.644 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:44.644 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:44Z" 2024-06-15T10:14:44.644 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:44.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:49.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:49.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:49.646 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:49.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:49.683 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:49.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:49.742+0000 7ffbb2866a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:49.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:49.742+0000 7ffbb2866a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:49.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:49.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:49.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:49.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:49.775 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:49Z" 2024-06-15T10:14:49.775 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:49.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:54.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:54.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:54.777 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:54.821 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:54.821 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:54.881 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:54.878+0000 7f226a849a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:54.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:54.878+0000 7f226a849a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:54.901 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:14:54.902 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:54.903 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:14:54.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:14:54.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:14:54Z" 2024-06-15T10:14:54.906 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:14:54.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:14:59.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:59.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:14:59.909 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:14:59.953 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:14:59.953 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:14:59.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:59.994+0000 7fc292464a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:14:59.999 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:14:59.998+0000 7fc292464a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.021 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:00.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:00.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:00.025 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:00Z" 2024-06-15T10:15:00.026 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:00.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:15:05.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:05.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:05.028 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:05.070 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:05.071 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:05.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:05.130+0000 7fe458a8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:05.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:05.134+0000 7fe458a8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:05.159 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.160 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.161 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:05.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:05.164 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:05Z" 2024-06-15T10:15:05.165 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:05.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:15:10.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:10.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:10.167 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:10.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:10.210 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:10.253 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:10.250+0000 7f473276aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:10.254 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:10.254+0000 7f473276aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.273 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:10.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:10.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:10.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:10.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:10.278 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:10Z" 2024-06-15T10:15:10.278 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:10.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:15:15.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:15.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:15.280 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:15.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:15.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:15.387 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:15.390+0000 7f663cfa6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:15.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:15.390+0000 7f663cfa6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:15.411 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:15.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:15.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:15.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:15.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:15Z" 2024-06-15T10:15:15.416 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:15.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:15:20.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:20.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:20.419 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:20.461 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:20.462 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:20.508 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:20.510+0000 7f9792a61a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:20.510 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:20.514+0000 7f9792a61a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:20.531 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:20.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:20.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:20.534 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:20.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:20.537 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:20Z" 2024-06-15T10:15:20.537 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:20.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:15:25.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:25.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:25.539 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:25.583 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:25.583 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:25.632 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:25.633+0000 7f35505a9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:25.633 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:25.633+0000 7f35505a9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:25.656 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:25.656 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:25.656 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:25.656 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:25.656 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:25.657 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:25.658 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:15:25.659 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:25.660 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:25Z" 2024-06-15T10:15:25.661 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:25.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:15:30.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:30.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:30.664 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:30.722 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:30.722 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:30.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:30.805+0000 7fb3fda58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:30.806 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:30.809+0000 7fb3fda58a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:30.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:30.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:30.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:30.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:30.836 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:30Z" 2024-06-15T10:15:30.837 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:30.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:15:35.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:35.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:35.840 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:35.884 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:35.884 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:35.928 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:35.929+0000 7f7735bb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:35.929 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:35.929+0000 7f7735bb6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:35.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:35.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.952 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:35.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:35Z" 2024-06-15T10:15:35.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:35.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:15:40.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:40.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:40.956 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:40.996 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:40.997 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:41.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:41.045+0000 7fe30037ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:41.045 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:41.045+0000 7fe30037ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:41.064 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:41.065 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:41.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:41.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:41.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:41Z" 2024-06-15T10:15:41.069 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:41.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:15:46.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:46.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:46.072 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:46.115 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:46.115 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:46.160 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:46.161+0000 7fde7adf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:46.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:46.161+0000 7fde7adf7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:46.179 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:46.179 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:46.179 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:46.179 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:46.179 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:46.180 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:46.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:15:46.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:46.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:46Z" 2024-06-15T10:15:46.184 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:46.185 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:15:51.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:51.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:51.187 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:51.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:51.231 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:51.288 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:51.289+0000 7f82b6df8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:51.289 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:51.289+0000 7f82b6df8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:51.310 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:51.310 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:51.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:51.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:51.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:51.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:51Z" 2024-06-15T10:15:51.315 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:51.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:15:56.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:56.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:15:56.318 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:15:56.372 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:15:56.373 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:15:56.429 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:56.429+0000 7f7db08eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:56.430 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:15:56.433+0000 7f7db08eaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:15:56.453 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:15:56.453 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:15:56.453 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:15:56.454 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:56.455 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:56.456 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:15:56.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:15:56Z" 2024-06-15T10:15:56.458 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:15:56.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:01.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:01.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:01.460 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:01.504 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:01.504 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:01.562 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:01.565+0000 7fcd048ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:01.563 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:01.565+0000 7fcd048ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:01.583 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:01.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:01.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:01.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:01.587 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:01Z" 2024-06-15T10:16:01.588 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:01.589 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:16:06.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:06.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:06.591 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:06.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:06.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:06.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:06.697+0000 7f07d3fc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:06.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:06.697+0000 7f07d3fc7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:06.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:06.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:06.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:06.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:06.725 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:06Z" 2024-06-15T10:16:06.725 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:06.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:16:11.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:11.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:11.727 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:11.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:11.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:11.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:11.821+0000 7fb0c7a81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:11.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:11.821+0000 7fb0c7a81a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:11.844 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:11.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:11.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:11.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:11.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:11.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:11Z" 2024-06-15T10:16:11.849 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:11.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:16.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:16.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:16.852 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:16.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:16.896 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:16.947 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:16.949+0000 7f6e290b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:16.948 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:16.949+0000 7f6e290b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:16.968 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:16.968 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:16.968 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:16.968 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:16.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:16.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:16Z" 2024-06-15T10:16:16.973 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:16.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:21.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:21.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:21.976 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:22.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:22.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:22.082 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:22.080+0000 7fc363348a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:22.083 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:22.084+0000 7fc363348a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:22.103 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:22.104 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:22.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:22.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:22.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:22Z" 2024-06-15T10:16:22.108 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:22.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:27.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:27.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:27.111 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:27.155 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:27.156 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:27.212 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:27.212+0000 7f035d239a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:27.213 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:27.212+0000 7f035d239a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:27.233 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:27.234 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:27.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:27.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:27.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:27Z" 2024-06-15T10:16:27.238 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:27.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:16:32.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:32.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:32.240 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:32.284 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:32.284 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:32.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:32.332+0000 7fa214206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:32.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:32.332+0000 7fa214206a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:32.359 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:32.359 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:32.359 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:32.359 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:32.359 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:32.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:32.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:32.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:32.363 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:32Z" 2024-06-15T10:16:32.364 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:32.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:37.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:37.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:37.367 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:37.410 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:37.410 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:37.454 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:37.452+0000 7f256c5f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:37.455 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:37.456+0000 7f256c5f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:37.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:37.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:37.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:37.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:16:37.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:37.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:37.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:37.480 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:37.481 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:37.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:37.482 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:37Z" 2024-06-15T10:16:37.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:37.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:16:42.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:42.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:42.484 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:42.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:42.529 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:42.586 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:42.584+0000 7fb9ceb75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:42.587 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:42.588+0000 7fb9ceb75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:42.605 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:42.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:42.607 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:42.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:42.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:42Z" 2024-06-15T10:16:42.610 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:42.611 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:16:47.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:47.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:47.613 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:47.653 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:47.654 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:47.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:47.696+0000 7efe04eeda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:47.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:47.696+0000 7efe04eeda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:47.714 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:47.715 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:47.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:47.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:47.719 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:47Z" 2024-06-15T10:16:47.719 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:47.720 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:16:52.721 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... FAIL 2024-06-15T10:16:52.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:52.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:52.722 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:52.863 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:16:52.886 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:16:52.887 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:16:52.888 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:16:52.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:52.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:52.889 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:52.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:52.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:52.971 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:52.972+0000 7f2965705a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:52.972 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:52.972+0000 7f2965705a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:52.993 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:52.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:52.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:52.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:53.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:53.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:53.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:53.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:53.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:53.002 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:53.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:53.002 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:52Z" 2024-06-15T10:16:53.002 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:53.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:16:58.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:58.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:16:58.005 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:16:58.050 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:16:58.050 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:16:58.108 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:58.108+0000 7fe92a0afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:58.109 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:16:58.108+0000 7fe92a0afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:16:58.140 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:16:58.140 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:16:58.140 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.141 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:16:58.142 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:58.143 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:16:58.144 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.145 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:58.146 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:16:58.146 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:58.146 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:58.146 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:16:58.146 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:58.147 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:16:58.148 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:58.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:16:58.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:16:58.151 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:16:58Z" 2024-06-15T10:16:58.151 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:16:58.152 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:03.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:03.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:03.153 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:03.197 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:03.198 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:03.248 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:03.248+0000 7f6246098a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:03.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:03.248+0000 7f6246098a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:03.273 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:03.273 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:03.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:03.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:03.276 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:03.277 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:03.278 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:03Z" 2024-06-15T10:17:03.279 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:03.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:17:08.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:08.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:08.282 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:08.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:08.324 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:08.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:08.372+0000 7fcf3181aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:08.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:08.372+0000 7fcf3181aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.391 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:08.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:08.393 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:08.394 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:08.395 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:08Z" 2024-06-15T10:17:08.396 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:08.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:17:13.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:13.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:13.398 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:13.442 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:13.442 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:13.488 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:13.487+0000 7f6f98140a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:13.489 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:13.487+0000 7f6f98140a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:13.507 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:13.507 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:13.507 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:13.507 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:13.507 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:13.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:13Z" 2024-06-15T10:17:13.512 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:13.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:17:18.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:18.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:18.514 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:18.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:18.558 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:18.623 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:18.623+0000 7f13bf46fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:18.624 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:18.623+0000 7f13bf46fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:18.650 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:18.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:18.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:18.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:18.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:18Z" 2024-06-15T10:17:18.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:18.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:23.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:23.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:23.657 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:23.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:23.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:23.742 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:23.739+0000 7f3640ce2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:23.743 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:23.743+0000 7f3640ce2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:23.761 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:23.762 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:23.763 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:23.764 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:23.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:23.766 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:23.767 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:23.767 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:23Z" 2024-06-15T10:17:23.767 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:23.768 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:28.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:28.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:28.769 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:28.812 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:28.813 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:28.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:28.859+0000 7f85da44aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:28.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:28.859+0000 7f85da44aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:28.879 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:28.879 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:28.879 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:28.880 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:28.881 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.107383_583.1", 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:28.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.107383Z", 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:28.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:28Z" 2024-06-15T10:17:28.884 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:28.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.107383_583.1 2024-06-15T10:17:33.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:33.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:33.887 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:33.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:33.930 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:33.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:33.971+0000 7fa6b5689a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:33.973 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:33.971+0000 7fa6b5689a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:33.994 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:33.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:33.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:33.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:33Z" 2024-06-15T10:17:33.999 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:34.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:39.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:39.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:39.002 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:39.044 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:39.045 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:39.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:39.091+0000 7fecb2345a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:39.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:39.091+0000 7fecb2345a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:39Z" 2024-06-15T10:17:39.116 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:39.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:17:44.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:44.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:44.119 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:44.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:44.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:44.213 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:44.211+0000 7fadb6c23a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:44.214 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:44.215+0000 7fadb6c23a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:44.235 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:44.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:44.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:44.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:44.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:44.240 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:44.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:44.241 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:44Z" 2024-06-15T10:17:44.241 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:44.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:17:49.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:49.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:49.244 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:49.286 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:49.287 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:49.342 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:49.343+0000 7fc22f821a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:49.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:49.343+0000 7fc22f821a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:49.376 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:49.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:49.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:49.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.379 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:49.380 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:49.381 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:49.382 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:49Z" 2024-06-15T10:17:49.383 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:49.384 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:54.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:54.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:54.386 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:54.430 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:54.431 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:54.475 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:54.475+0000 7f16e8725a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:54.475 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:54.475+0000 7f16e8725a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:54.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:54.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:54.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:54.505 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:54.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:54.505 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:54Z" 2024-06-15T10:17:54.505 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:54.506 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:17:59.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:59.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:17:59.508 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:17:59.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:17:59.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:17:59.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:59.591+0000 7fa97eaefa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:59.593 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:17:59.591+0000 7fa97eaefa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.612 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:59.613 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:17:59.614 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:17:59.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:17:59.616 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:17:59Z" 2024-06-15T10:17:59.617 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:17:59.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:18:04.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:04.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:04.619 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:04.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:04.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:04.709 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:04.706+0000 7fcbf1adca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:04.710 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:04.710+0000 7fcbf1adca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:04.734 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:04.735 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:04.736 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:04.737 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:04.738 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:04Z" 2024-06-15T10:18:04.739 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:04.739 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:18:09.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:09.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:09.742 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:09.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:09.822+0000 7f7a42075a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:09.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:09.822+0000 7f7a42075a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:09.843 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:09.843 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:09.843 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:09.843 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:09.843 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:09.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:09.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:18:09.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:09.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:09Z" 2024-06-15T10:18:09.848 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:09.849 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:18:14.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:14.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:14.851 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:14.904 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:14.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:14.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:14.974+0000 7f7e0ba90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:14.976 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:14.974+0000 7f7e0ba90a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:14.995 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:14.995 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:14.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:14.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:14.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:14.999 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:14Z" 2024-06-15T10:18:15.000 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:15.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:18:20.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:20.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:20.002 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:20.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:20.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:20.097 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:20.094+0000 7f29bcc20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:20.098 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:20.098+0000 7f29bcc20a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:20.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:20.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:20.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:20.123 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:20.124 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:20Z" 2024-06-15T10:18:20.125 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:20.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:18:25.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:25.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:25.128 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:25.172 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:25.172 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:25.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:25.246+0000 7f5e8e078a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:25.250 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:25.250+0000 7f5e8e078a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:25.289 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:25.289 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:25.289 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:25.289 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:25.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:25.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440137.061971_914.1", 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:57.061971Z", 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:25.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:25.293 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:25Z" 2024-06-15T10:18:25.294 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:25.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440137.061971_914.1 2024-06-15T10:18:30.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:30.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:30.296 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:30.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:30.339 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:30.388 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:30.386+0000 7fb4fae79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:30.389 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:30.386+0000 7fb4fae79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:30.410 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:30.410 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:30.411 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:30.412 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:18:30.413 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:30.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:30Z" 2024-06-15T10:18:30.415 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:30.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:18:35.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:35.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:35.417 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:35.462 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:35.462 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:35.513 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:35.510+0000 7fd2780baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:35.513 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:35.510+0000 7fd2780baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:35.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:35.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:35.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:35Z" 2024-06-15T10:18:35.543 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:35.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:18:40.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:40.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:40.546 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:40.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:40.602 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:40.654 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:40.654+0000 7faeb6099a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:40.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:40.654+0000 7faeb6099a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:40.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:40.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:40.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:40.678 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:40.679 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:40.679 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:40Z" 2024-06-15T10:18:40.679 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:40.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:18:45.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:45.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:45.681 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:45.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:45.727 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:45.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:45.786+0000 7f265af6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:45.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:45.786+0000 7f265af6da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:45.808 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:45.809 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:45.810 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:45.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:45.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:45Z" 2024-06-15T10:18:45.813 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:45.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:18:50.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:50.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:50.816 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:50.858 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:50.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:50.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:50.905+0000 7fdc79a72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:50.905 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:50.905+0000 7fdc79a72a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:50.932 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:50.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:50.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:50.935 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:50.936 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:50.937 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:50Z" 2024-06-15T10:18:50.938 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:50.938 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:18:55.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:55.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:18:55.940 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:18:55.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:18:55.983 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:18:56.030 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:56.033+0000 7ffbf457aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:56.031 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:18:56.033+0000 7ffbf457aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:18:56.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:56.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.052 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:18:56.053 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.054 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:18:56.055 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:18:56.055 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:18:56.055 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:18:56.055 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:18:56Z" 2024-06-15T10:18:56.055 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:18:56.056 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:01.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:01.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:01.057 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:01.107 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:01.107 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:01.151 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:01.153+0000 7f0139163a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:01.152 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:01.153+0000 7f0139163a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:01.171 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:01.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:19:01.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:01.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:01Z" 2024-06-15T10:19:01.175 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:01.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:19:06.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:06.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:06.178 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:06.224 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:06.224 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:06.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:06.285+0000 7fc162f37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:06.285 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:06.285+0000 7fc162f37a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:06.305 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:06.305 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:06.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:06.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:06.308 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:06.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:06Z" 2024-06-15T10:19:06.310 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:06.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:11.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:11.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:11.313 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:11.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:11.356 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:11.400 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:11.401+0000 7fe648db9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:11.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:11.401+0000 7fe648db9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:11.420 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:11.421 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:11.422 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:11.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:11.424 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:11Z" 2024-06-15T10:19:11.425 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:11.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:16.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:16.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:16.427 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:16.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:16.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:16.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:16.521+0000 7f47dff9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:16.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:16.525+0000 7f47dff9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:16.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:16.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:16.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:16Z" 2024-06-15T10:19:16.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:16.549 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:21.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:21.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:21.551 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:21.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:21.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:21.655 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:21.657+0000 7f13704b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:21.656 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:21.657+0000 7f13704b7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:21.682 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:21.682 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:21.682 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:21.682 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:21.682 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:21.683 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:21.684 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:21.685 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:21.686 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:21Z" 2024-06-15T10:19:21.687 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:21.688 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:26.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:26.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:26.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:26.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:26.733 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:26.778 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:26.777+0000 7fbc4c629a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:26.778 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:26.781+0000 7fbc4c629a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:26.798 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:26.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:26.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.799 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:26.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:26.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:26.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:26.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:26.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:26.803 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:26.804 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:26.804 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:26.804 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:26.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.805 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:26.806 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:26.806 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:26.806 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:26.806 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:26.806 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:26.807 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:26.807 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:26.807 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:26.807 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:26.807 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:26.808 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:26Z" 2024-06-15T10:19:26.809 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:26.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:31.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:31.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:31.812 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:31.866 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:31.867 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:31.946 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:31.949+0000 7f5c209d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:31.947 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:31.949+0000 7f5c209d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:31.965 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:31.965 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:31.965 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:31.965 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:31.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:31.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:31.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:31.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:31.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:31.972 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:31.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:31.973 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:31Z" 2024-06-15T10:19:31.973 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:31.974 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:19:36.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:36.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:36.975 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:37.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:37.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:37.063 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:37.065+0000 7fe6c9f7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:37.064 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:37.065+0000 7fe6c9f7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:37.083 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:37.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:37.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:37.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:37Z" 2024-06-15T10:19:37.088 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:37.089 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:19:42.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:42.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:42.091 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:42.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:42.135 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:42.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:42.188+0000 7fda4c9fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:42.188 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:42.188+0000 7fda4c9fea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:42.206 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:42.207 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:42.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:42.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:42.210 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:42.211 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:42Z" 2024-06-15T10:19:42.212 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:42.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:19:47.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:47.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:47.214 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:47.257 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:47.257 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:47.302 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:47.300+0000 7f970f3b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:47.303 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:47.304+0000 7f970f3b3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:47.325 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:47.325 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:47.325 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:47.326 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:47.326 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:47.326 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:47.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:47.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:47.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:47.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439520.005312_808.1", 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:47.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:18:40.005312Z", 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:47.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:47Z" 2024-06-15T10:19:47.331 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:47.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718439520.005312_808.1 2024-06-15T10:19:52.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:52.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:52.334 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:52.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:52.377 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:52.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:52.424+0000 7fc721d1ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:52.426 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:52.424+0000 7fc721d1ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:52.444 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:52.444 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:52.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:52.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.769917_671.1", 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:52.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.769917Z", 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:52.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:52.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:52.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:52Z" 2024-06-15T10:19:52.450 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:52.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.769917_671.1 2024-06-15T10:19:57.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:57.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:19:57.453 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:19:57.496 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:19:57.496 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:19:57.551 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:57.552+0000 7f558bc9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:57.552 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:19:57.552+0000 7f558bc9ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:19:57.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:57.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:19:57.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:19:57.578 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:19:57.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:19:57.580 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:19:57Z" 2024-06-15T10:19:57.580 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:19:57.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:20:02.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:02.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:02.582 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:02.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:02.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:02.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:02.672+0000 7f4c75c6ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:02.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:02.672+0000 7f4c75c6ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:02.691 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.692 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:02.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.806995_673.1", 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.806995Z", 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:02.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.695 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:02.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:02.696 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:02.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:02.696 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:02Z" 2024-06-15T10:20:02.696 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:02.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.806995_673.1 2024-06-15T10:20:07.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:07.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:07.698 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:07.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:07.748 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:07.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:07.812+0000 7f85f63b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:07.814 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:07.816+0000 7f85f63b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:07.833 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:07.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:07.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:07.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:07Z" 2024-06-15T10:20:07.839 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:07.840 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:20:12.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:12.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:12.841 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:12.885 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:12.885 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:12.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:12.936+0000 7fa63484da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:12.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:12.940+0000 7fa63484da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:12.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:12.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:12.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.962 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:12.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:12.963 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:12.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:12.963 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:12Z" 2024-06-15T10:20:12.963 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:12.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:20:17.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:17.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:17.966 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:18.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:18.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:18.058 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:18.056+0000 7ffa3a95ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:18.059 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:18.060+0000 7ffa3a95ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:18.079 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:18.080 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:18.081 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:18.082 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:18Z" 2024-06-15T10:20:18.083 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:18.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:20:23.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:23.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:23.085 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:23.128 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:23.129 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:23.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:23.176+0000 7f7c8ff4ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:23.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:23.180+0000 7f7c8ff4ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:23.197 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:23.197 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:23.198 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:23.199 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:23.200 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:23.201 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:23Z" 2024-06-15T10:20:23.202 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:23.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:20:28.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:28.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:28.205 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:28.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:28.248 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:28.296 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:28.296+0000 7f1681681a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:28.297 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:28.296+0000 7f1681681a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:28.316 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:28.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:28.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:28.319 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:28.320 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:28Z" 2024-06-15T10:20:28.321 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:28.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:20:33.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:33.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:33.324 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:33.368 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:33.368 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:33.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:33.411+0000 7f327990ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:33.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:33.411+0000 7f327990ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:33.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:33.438 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:33.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:33.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:33.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:33.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:33.443 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:33.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:33.443 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:33Z" 2024-06-15T10:20:33.443 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:33.444 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.915314_676.1 2024-06-15T10:20:38.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:38.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:38.445 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:38.489 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:38.490 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:38.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:38.539+0000 7fdbbc6dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:38.539 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:38.539+0000 7fdbbc6dda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:38.558 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:38.558 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:38.558 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:38.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:38.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:38.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:38.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:38Z" 2024-06-15T10:20:38.564 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:38.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:20:43.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:43.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:43.566 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:43.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:43.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:43.670 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:43.667+0000 7f0985d3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:43.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:43.671+0000 7f0985d3da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:43.697 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:43.698 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:43.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:43.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:43.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:43Z" 2024-06-15T10:20:43.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:43.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:20:48.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:48.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:48.704 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:48.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:48.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:48.808 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:48.807+0000 7fab233a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:48.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:48.807+0000 7fab233a2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:48.832 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:48.832 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:48.832 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:48.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:48.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:48.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:48.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:48Z" 2024-06-15T10:20:48.837 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:48.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:20:53.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:53.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:53.840 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:53.882 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:53.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:53.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:53.927+0000 7f2516dc3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:53.927 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:53.927+0000 7f2516dc3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:53.947 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:53.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:53.949 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:53.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:53.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:53Z" 2024-06-15T10:20:53.952 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:53.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:20:58.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:58.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:20:58.955 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:20:58.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:20:58.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:20:59.040 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:59.039+0000 7f602e498a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:59.041 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:20:59.039+0000 7f602e498a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:20:59.065 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:59.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:59.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:20:59.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:20:59.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:20:59.070 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:20:59.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:20:59.070 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:20:59Z" 2024-06-15T10:20:59.070 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:20:59.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:04.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:04.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:04.074 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:04.117 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:04.118 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:04.164 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:04.163+0000 7fe6957d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:04.165 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:04.163+0000 7fe6957d9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:04.184 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:04.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:04.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:21:04.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:04.188 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:04Z" 2024-06-15T10:21:04.189 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:04.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:21:09.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:09.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:09.192 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:09.235 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:09.236 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:09.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:09.275+0000 7f83935eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:09.279 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:09.279+0000 7f83935eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:09.298 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:09.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:09.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:09.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:09.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:09.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:09.304 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:09Z" 2024-06-15T10:21:09.304 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:09.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:14.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:14.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:14.310 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:14.358 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:14.358 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:14.412 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:14.411+0000 7fb238d7ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:14.413 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:14.411+0000 7fb238d7ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:14.439 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:14.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:14.441 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:14.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:14.443 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:14Z" 2024-06-15T10:21:14.444 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:14.445 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:21:19.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:19.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:19.446 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:19.491 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:19.491 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:19.541 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:19.539+0000 7f01542bca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:19.542 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:19.543+0000 7f01542bca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:19.563 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:19.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.566 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:19.567 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:19Z" 2024-06-15T10:21:19.568 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:19.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:24.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:24.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:24.570 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:24.615 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:24.616 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:24.674 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:24.670+0000 7f6d2ecb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:24.675 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:24.674+0000 7f6d2ecb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:24.700 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:24.700 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:24.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:24.702 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:24.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:24.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:24Z" 2024-06-15T10:21:24.705 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:24.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:29.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:29.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:29.708 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:29.760 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:29.760 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:29.809 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:29.806+0000 7f43b920ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:29.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:29.810+0000 7f43b920ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:29.829 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:29.829 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:29.829 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:29.829 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:29.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:29.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:29.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:29.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:29Z" 2024-06-15T10:21:29.834 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:29.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:34.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:34.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:34.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:34.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:34.879 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:34.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:34.934+0000 7f6365088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:34.937 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:34.934+0000 7f6365088a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:34Z" 2024-06-15T10:21:34.964 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:34.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:21:39.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:39.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:39.967 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:40.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:40.014 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:40.067 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:40.066+0000 7f619d37fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:40.068 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:40.066+0000 7f619d37fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:40.090 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:40.095 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:40Z" 2024-06-15T10:21:40.096 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:40.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:21:45.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:45.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:45.098 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:45.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:45.143 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:45.215 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:45.214+0000 7f2231eb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:45.216 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:45.214+0000 7f2231eb0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:45.233 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:45.233 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:45.233 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:45.233 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:45.234 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:45.235 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:45.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:45.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:45Z" 2024-06-15T10:21:45.238 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:45.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:50.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:50.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:50.242 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:50.285 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:50.285 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:50.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:50.326+0000 7f3b86534a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:50.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:50.326+0000 7f3b86534a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:50.352 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:50.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:50.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.356 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:50.357 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:50.357 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:50.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:50.357 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:50Z" 2024-06-15T10:21:50.357 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:50.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:21:55.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:55.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:21:55.359 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:21:55.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:21:55.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:21:55.452 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:55.450+0000 7fbdadea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:55.452 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:21:55.450+0000 7fbdadea4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:21:55.480 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:55.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.482 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:21:55.483 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:21:55.484 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:21:55Z" 2024-06-15T10:21:55.485 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:21:55.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:22:00.487 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... FAIL 2024-06-15T10:22:00.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:00.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:00.488 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:00.548 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:00.549 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:00.628 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:22:00.647 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:22:00.647 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:22:00.647 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:00.647 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:22:00.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:22:00.648 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:22:00.649 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:22:00.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:00.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:00.650 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:00.694 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:00.694 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:00.751 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:00.750+0000 7f97e0924a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:00.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:00.750+0000 7f97e0924a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:00.772 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:00.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:00.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:22:00.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:00.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:00Z" 2024-06-15T10:22:00.777 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:00.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:22:05.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:05.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:05.779 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:05.822 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:05.822 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:05.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:05.866+0000 7fe973880a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:05.866 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:05.870+0000 7fe973880a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:05.885 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:05.886 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:05.887 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:05.888 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:05Z" 2024-06-15T10:22:05.889 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:05.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:10.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:10.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:10.892 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:10.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:10.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:10.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:10.997+0000 7feea79d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:10.998 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:11.001+0000 7feea79d7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:11.019 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:11.019 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:11.020 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:11.022 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:11.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:11Z" 2024-06-15T10:22:11.024 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:11.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:16.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:16.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:16.027 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:16.070 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:16.071 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:16.114 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:16.113+0000 7ff0b86cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:16.115 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:16.117+0000 7ff0b86cba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:16.134 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:16.135 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:16.136 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:16.137 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:16Z" 2024-06-15T10:22:16.138 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:16.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:22:21.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:21.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:21.140 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:21.184 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:21.184 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:21.229 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:21.229+0000 7f4f374dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:21.230 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:21.229+0000 7f4f374dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:21.250 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:21.251 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:21.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:21.253 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.254 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:21.255 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:21.255 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:21.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:21.255 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:21Z" 2024-06-15T10:22:21.255 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:21.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:22:26.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:26.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:26.258 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:26.308 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:26.309 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:26.365 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:26.365+0000 7f52f74f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:26.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:26.365+0000 7f52f74f2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:26.384 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:26.385 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:26.385 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:26.385 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:26.385 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:26.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:26.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:26.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:26.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:26.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:26.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:26.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:26.393 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:26.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:26.393 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:26Z" 2024-06-15T10:22:26.393 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:26.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:31.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:31.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:31.395 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:31.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:31.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:31.501 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:31.501+0000 7f7433cf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:31.502 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:31.505+0000 7f7433cf5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:31.528 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:31.528 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:31.528 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:31.528 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:31.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:31.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:31.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:31Z" 2024-06-15T10:22:31.533 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:31.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:36.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:36.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:36.535 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:36.581 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:36.581 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:36.629 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:36.629+0000 7f24c76bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:36.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:36.633+0000 7f24c76bda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:36.650 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:36.650 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:36.650 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:36.650 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:36.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:36.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:36.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:36.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:36.654 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:36Z" 2024-06-15T10:22:36.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:36.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:41.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:41.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:41.658 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:41.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:41.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:41.765 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:41.765+0000 7ff539329a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:41.766 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:41.769+0000 7ff539329a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:41.809 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.810 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.811 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:41.812 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:41.813 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:41.814 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:41.815 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:41.815 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:41Z" 2024-06-15T10:22:41.815 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:41.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:46.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:46.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:46.817 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:46.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:46.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:46.909 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:46.909+0000 7f6820d45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:46.910 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:46.913+0000 7f6820d45a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:46.930 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.931 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.932 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:46.933 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:46.934 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:46Z" 2024-06-15T10:22:46.935 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:46.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:22:51.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:51.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:51.937 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:51.980 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:51.980 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:52.040 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:52.041+0000 7f985a601a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:52.041 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:52.041+0000 7f985a601a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:52.068 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:52.069 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:52.070 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:52.071 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:22:52.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:52.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:52.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:52Z" 2024-06-15T10:22:52.075 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:52.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:22:57.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:57.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:22:57.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:22:57.136 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:22:57.137 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:22:57.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:57.189+0000 7f49d8688a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:57.188 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:22:57.189+0000 7f49d8688a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:22:57.209 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:22:57.209 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:22:57.209 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:22:57.209 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:22:57.209 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:22:57.210 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:22:57.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:57.210 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.210 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:22:57.211 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:57.211 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:22:57.211 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:57.211 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:57.211 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:22:57.212 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:57.212 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:22:57.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:57.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:22:57.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:57.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:57.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.213 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:57.213 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:22:57.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:57.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:57.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:22:57.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:57.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:57.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:57.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:22:57.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:57.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:57.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:22:57.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:57.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:57.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:57.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:22:57.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:57.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:57.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:22:57.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:22:57.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:22:57.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:22:57.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:22:57.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:57.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:22:57.220 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:22:57.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:22:57.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:22:57.221 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:22:57.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:22:57.221 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:22:57Z" 2024-06-15T10:22:57.221 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:22:57.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:02.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:02.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:02.224 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:02.267 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:02.268 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:02.326 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:02.328+0000 7f56d6a03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:02.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:02.328+0000 7f56d6a03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:02.346 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:02.347 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:02.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:02.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:02.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:02Z" 2024-06-15T10:23:02.351 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:02.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:23:07.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:07.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:07.354 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:07.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:07.398 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:07.443 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:07.444+0000 7f6d49aa1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:07.444 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:07.444+0000 7f6d49aa1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.463 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:07.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:07.466 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:07.467 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:07Z" 2024-06-15T10:23:07.468 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:07.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:23:12.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:12.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:12.469 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:12.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:12.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:12.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:12.580+0000 7f0e5d861a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:12.579 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:12.580+0000 7f0e5d861a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:12.600 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:12.601 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:12.602 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:12.603 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:12.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:12.605 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:12Z" 2024-06-15T10:23:12.606 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:12.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:17.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:17.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:17.608 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:17.654 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:17.654 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:17.698 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:17.696+0000 7f008f9a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:17.699 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:17.700+0000 7f008f9a5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:17.718 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:17.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:17.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:23:17.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:17.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:17Z" 2024-06-15T10:23:17.723 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:17.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:23:22.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:22.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:22.725 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:22.766 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:22.766 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:22.810 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:22.808+0000 7f2f7b9eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:22.811 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:22.812+0000 7f2f7b9eea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:22.829 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:22.829 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:22.829 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:22.829 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:22.829 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:22.830 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:22.831 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:22.832 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:22.833 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:22Z" 2024-06-15T10:23:22.834 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:22.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:27.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:27.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:27.837 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:27.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:27.881 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:27.929 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:27.928+0000 7f3272b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:27.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:27.932+0000 7f3272b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:27.949 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:27.950 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:27.951 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:27.952 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:27Z" 2024-06-15T10:23:27.953 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:27.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:32.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:32.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:32.956 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:33.000 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:33.049 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:33.048+0000 7f192329fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:33.050 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:33.052+0000 7f192329fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:33.070 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:33.071 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:33.074 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:33Z" 2024-06-15T10:23:33.075 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:33.076 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:38.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:38.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:38.078 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:38.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:38.120 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:38.180 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:38.180+0000 7f25ca567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:38.181 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:38.180+0000 7f25ca567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:38.206 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:38.206 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:38.206 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:38.206 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:38.207 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:38.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:38.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:38.210 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:38Z" 2024-06-15T10:23:38.211 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:38.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:23:43.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:43.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:43.213 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:43.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:43.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:43.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:43.320+0000 7f5cc36dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:43.319 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:43.320+0000 7f5cc36dca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:43.339 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:43.339 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:43.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:43.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:43.342 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:43.343 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:43Z" 2024-06-15T10:23:43.344 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:43.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:48.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:48.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:48.346 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:48.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:48.389 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:48.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:48.440+0000 7f3b5e09aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:48.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:48.440+0000 7f3b5e09aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:48.459 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:48.460 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:48.461 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:48.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:48.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:48.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:48Z" 2024-06-15T10:23:48.466 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:48.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:53.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:53.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:53.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:53.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:53.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:53.551 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:53.551+0000 7f59faf5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:53.552 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:53.551+0000 7f59faf5ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:53.572 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:53.573 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:53.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:53.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:53.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:53.577 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:53.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:53.578 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:53Z" 2024-06-15T10:23:53.578 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:53.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:23:58.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:58.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:23:58.580 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:23:58.622 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:23:58.623 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:23:58.685 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:58.683+0000 7f0ab25baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:58.686 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:23:58.687+0000 7f0ab25baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:23:58.715 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:23:58.715 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:23:58.716 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:23:58.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:58.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:23:58.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:23:58Z" 2024-06-15T10:23:58.721 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:23:58.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:24:03.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:03.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:03.724 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:03.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:03.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:03.819 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:03.819+0000 7fdfd85fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:03.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:03.819+0000 7fdfd85fca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:03.840 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:03.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:03.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:03.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.844 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:03.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:03.845 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:03.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:03.845 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:03Z" 2024-06-15T10:24:03.845 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:03.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:24:08.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:08.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:08.847 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:08.890 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:08.890 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:08.944 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:08.943+0000 7f10a1432a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:08.945 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:08.943+0000 7f10a1432a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:08.970 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:08.970 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:08.970 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:08.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:08.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:08.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:08.974 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:08Z" 2024-06-15T10:24:08.975 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:08.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:24:13.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:13.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:13.978 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:14.021 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:14.021 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:14.083 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:14.083+0000 7f0d2ff63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:14.084 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:14.083+0000 7f0d2ff63a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:14.104 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:14.105 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:14.106 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:14.107 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.108 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:14.109 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:14.109 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:14.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:14.109 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:14Z" 2024-06-15T10:24:14.109 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:14.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:24:19.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:19.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:19.111 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:19.165 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:19.165 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:19.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:19.247+0000 7f0266c89a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:19.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:19.247+0000 7f0266c89a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.269 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:19.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:19.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:19.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:19.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:19.274 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:19.274 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:19.274 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:19Z" 2024-06-15T10:24:19.274 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:19.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:24:24.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:24.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:24.276 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:24.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:24.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:24.365 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:24.363+0000 7fa917ee3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:24.365 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:24.363+0000 7fa917ee3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:24.386 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:24.386 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:24.386 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:24.386 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:24.386 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:24.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:24.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:24.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:24.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:24Z" 2024-06-15T10:24:24.391 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:24.392 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:24:29.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:29.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:29.394 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:29.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:29.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:29.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:29.487+0000 7fe0f6e64a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:29.487 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:29.487+0000 7fe0f6e64a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.510 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:29.512 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:29.514 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:29Z" 2024-06-15T10:24:29.515 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:29.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:24:34.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:34.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:34.517 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:34.566 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:34.566 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:34.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:34.623+0000 7f990649ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:34.626 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:34.627+0000 7f990649ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:34.645 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:34.645 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:34.645 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:34.645 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:34.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:34.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:34.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:34.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:34Z" 2024-06-15T10:24:34.650 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:34.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:24:39.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:39.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:39.653 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:39.696 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:39.696 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:39.752 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:39.751+0000 7f68f8567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:39.753 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:39.751+0000 7f68f8567a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:39.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:39.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:39.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:39.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:39.779 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:39.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:39.779 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:39Z" 2024-06-15T10:24:39.779 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:39.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:24:44.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:44.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:44.781 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:44.825 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:44.825 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:44.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:44.870+0000 7f03f2627a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:44.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:44.870+0000 7f03f2627a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:44.891 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:44.891 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:44.891 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:44.891 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:44.891 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:44.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:44.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:44.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:44.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:44Z" 2024-06-15T10:24:44.896 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:44.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:24:49.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:49.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:49.899 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:49.960 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:49.960 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:50.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:50.010+0000 7f878cfd7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:50.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:50.014+0000 7f878cfd7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:50.034 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:50.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:50.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:50.037 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:50.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:50Z" 2024-06-15T10:24:50.039 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:50.040 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:24:55.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:55.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:24:55.042 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:24:55.085 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:24:55.085 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:24:55.132 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:55.130+0000 7f08aeac5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:55.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:24:55.130+0000 7f08aeac5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:24:55.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:55.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:24:55.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:24:55.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:24:55.158 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:24:55.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:24:55.159 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:24:55Z" 2024-06-15T10:24:55.159 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:24:55.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:25:00.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:00.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:00.161 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:00.195 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:00.195 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:00.236 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:00.234+0000 7f6ad218aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:00.237 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:00.234+0000 7f6ad218aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:00.259 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:00.260 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:00.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:00.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:00.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:00.264 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:00.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:00.265 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:00Z" 2024-06-15T10:25:00.265 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:00.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:25:05.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:05.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:05.267 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:05.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:05.310 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:05.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:05.358+0000 7fde46b7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:05.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:05.358+0000 7fde46b7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:05.382 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:05.382 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:05.382 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:05.382 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:05.383 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:05.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:05.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:05.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:05.387 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:05Z" 2024-06-15T10:25:05.388 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:05.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:25:10.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:10.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:10.390 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:10.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:10.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:10.483 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:10.482+0000 7fcb858a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:10.484 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:10.482+0000 7fcb858a1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:10.505 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:10.506 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:10.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:10.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:10.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:10Z" 2024-06-15T10:25:10.511 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:10.512 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:25:15.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:15.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:15.513 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:15.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:15.577 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:15.625 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:15.622+0000 7fd520f04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:15.626 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:15.622+0000 7fd520f04a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:15.645 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:15.646 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:15.647 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.649 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:15.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:15.650 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:15.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:15.650 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:15Z" 2024-06-15T10:25:15.650 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:15.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:25:20.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:20.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:20.653 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:20.697 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:20.697 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:20.744 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:20.742+0000 7f05fca1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:20.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:20.742+0000 7f05fca1fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:20.764 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:20.765 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:20.766 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:20.767 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:20.767 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:20.767 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:20.767 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:20.767 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:20.768 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:20.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:20.770 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:20.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:20.771 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:20Z" 2024-06-15T10:25:20.771 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:20.772 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:25:25.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:25.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:25.774 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:25.818 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:25.818 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:25.867 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:25.870+0000 7fd4d990aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:25.868 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:25.870+0000 7fd4d990aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:25.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:25.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:25.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:25.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:25Z" 2024-06-15T10:25:25.895 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:25.896 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:25:30.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:30.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:30.898 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:30.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:30.941 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:31.003 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:31.006+0000 7f5ec81b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:31.004 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:31.006+0000 7f5ec81b4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:31.043 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:31.043 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:31.043 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:31.043 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:31.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:31.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:31.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:31.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:31Z" 2024-06-15T10:25:31.048 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:31.049 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:25:36.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:36.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:36.050 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:36.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:36.094 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:36.155 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:36.157+0000 7fde0da2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:36.158 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:36.161+0000 7fde0da2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:36.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:36.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:36.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:36.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:36.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:36Z" 2024-06-15T10:25:36.187 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:36.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:25:41.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:41.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:41.189 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:41.238 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:41.239 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:41.290 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:41.289+0000 7faf23056a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:41.291 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:41.293+0000 7faf23056a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:41.314 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:41.315 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:41.316 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.317 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:41.318 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:41Z" 2024-06-15T10:25:41.319 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:41.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:25:46.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:46.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:46.322 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:46.364 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:46.364 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:46.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:46.421+0000 7f7f87fbca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:46.422 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:46.425+0000 7f7f87fbca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:46.442 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:46.442 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:46.442 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:46.442 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:46.442 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:46.443 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:46.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:46.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:46.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:46Z" 2024-06-15T10:25:46.447 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:46.448 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:25:51.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:51.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:51.449 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:51.492 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:51.492 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:51.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:51.537+0000 7f9a711f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:51.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:51.541+0000 7f9a711f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.560 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:51.562 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:51.562 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:51.562 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:51.562 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:51.562 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:51.563 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:51.564 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:51.565 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:51Z" 2024-06-15T10:25:51.566 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:51.567 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:25:56.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:56.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:25:56.568 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:25:56.624 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:25:56.624 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:25:56.700 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:56.701+0000 7f9e25bf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:56.700 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:25:56.701+0000 7f9e25bf6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.722 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.723 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:56.724 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:25:56.725 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:25:56.726 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:25:56.727 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:25:56.727 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:25:56.727 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:25:56Z" 2024-06-15T10:25:56.727 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:25:56.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:26:01.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:01.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:01.729 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:01.772 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:01.772 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:01.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:01.829+0000 7f0a050b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:01.830 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:01.833+0000 7f0a050b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:01.851 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:01.851 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:01.851 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:01.851 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:01.851 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:01.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:01.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:01.854 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:01.855 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:01Z" 2024-06-15T10:26:01.856 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:01.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:26:06.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:06.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:06.859 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:06.903 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:06.904 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:06.963 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:06.965+0000 7f604d92da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:06.964 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:06.965+0000 7f604d92da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:06.989 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:06.989 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:06.990 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:06.991 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440137.070711_915.1", 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:06.992 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:57.070711Z", 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:06.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:06Z" 2024-06-15T10:26:06.994 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:06.995 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440137.070711_915.1 2024-06-15T10:26:11.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:11.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:11.996 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:12.041 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:12.041 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:12.100 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:12.101+0000 7f0b21471a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:12.101 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:12.101+0000 7f0b21471a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:12.124 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:12.124 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:12.125 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:12.126 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:12.127 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:12.128 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:12Z" 2024-06-15T10:26:12.129 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:12.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:26:17.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:17.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:17.132 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:17.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:17.174 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:17.230 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:17.229+0000 7fdf09776a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:17.231 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:17.233+0000 7fdf09776a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:17.250 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:17.250 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:17.250 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:17.250 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:17.251 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:17.252 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:17.253 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:17.254 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:17Z" 2024-06-15T10:26:17.255 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:17.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:26:22.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:22.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:22.257 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:22.299 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:22.300 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:22.343 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:22.345+0000 7ff5a74baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:22.344 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:22.345+0000 7ff5a74baa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:22.362 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:22.363 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:22.364 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:22.365 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:22.366 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:22Z" 2024-06-15T10:26:22.367 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:22.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:26:27.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:27.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:27.370 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:27.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:27.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:27.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:27.468+0000 7fe99733aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:27.469 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:27.468+0000 7fe99733aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:27.488 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:27.489 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:27.490 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:27.492 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:27Z" 2024-06-15T10:26:27.493 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:27.494 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:26:32.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:32.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:32.495 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:32.539 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:32.539 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:32.591 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:32.592+0000 7fadd0e8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:32.592 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:32.592+0000 7fadd0e8ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:32.615 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:32.615 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:32.615 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:32.615 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:32.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:32.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:32.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:32.619 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:32Z" 2024-06-15T10:26:32.620 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:32.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:26:37.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:37.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:37.622 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:37.666 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:37.666 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:37.716 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:37.716+0000 7f6889aa4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:37.717 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:37.716+0000 7f6889aa4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:37.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:37.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:37.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:37.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:37Z" 2024-06-15T10:26:37.743 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:37.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:26:42.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:42.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:42.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:42.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:42.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:42.833 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:42.832+0000 7fd1a28efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:42.834 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:42.832+0000 7fd1a28efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:42.852 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:42.852 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:42.852 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:42.853 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:42.854 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:42.855 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:26:42.856 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:42.857 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:42.858 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:42.859 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:42.859 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:42Z" 2024-06-15T10:26:42.859 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:42.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:26:47.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:47.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:47.862 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:47.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:47.905 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:47.948 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:47.948+0000 7fa911796a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:47.948 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:47.948+0000 7fa911796a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:47.969 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:47.969 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:47.969 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:47.969 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:47.969 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:47.970 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:47.971 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:47.972 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:47.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:47Z" 2024-06-15T10:26:47.974 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:47.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:26:52.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:52.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:52.977 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:53.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:53.020 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:53.082 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:53.080+0000 7fc5da179a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:53.083 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:53.084+0000 7fc5da179a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:53.108 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:53.108 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:53.108 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:53.108 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:53.108 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:53.109 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:53.109 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:53.109 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.109 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:53.109 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:53.110 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:53.110 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:53.110 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:53.110 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:53.111 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:53.112 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:53.113 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:53.113 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:53.113 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:53.113 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:53.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:53.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:53.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:53.116 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:53.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:53.117 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:53Z" 2024-06-15T10:26:53.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:53.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:26:58.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:58.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:26:58.119 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:26:58.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:26:58.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:26:58.213 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:58.212+0000 7f9082b13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:58.214 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:26:58.216+0000 7f9082b13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:26:58.234 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:26:58.234 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:26:58.234 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:26:58.234 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:26:58.234 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:58.235 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:26:58.236 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:26:58.237 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:26:58.238 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:26:58.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:26:58.241 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:26:58.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:26:58.241 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:26:58Z" 2024-06-15T10:26:58.241 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:26:58.242 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:27:03.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:03.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:03.243 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:03.286 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:03.287 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:03.332 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:03.332+0000 7feff9a5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:03.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:03.332+0000 7feff9a5ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:03.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:03.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:03.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:03.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:03.357 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:03.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:03.357 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:03Z" 2024-06-15T10:27:03.357 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:03.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:27:08.359 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2024-06-15T10:27:08.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:08.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:08.360 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:08.405 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:08.458 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:08.478 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:27:08.479 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:27:08.479 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:27:08.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:08.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:08.480 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:08.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:08.522 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:08.565 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:08.564+0000 7f20a8932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:08.567 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:08.568+0000 7f20a8932a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:08.592 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:08.593 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.594 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:08.595 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:08.596 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:08.597 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:08.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:08.597 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:08Z" 2024-06-15T10:27:08.597 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:08.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:27:13.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:13.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:13.599 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:13.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:13.642 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:13.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:13.695+0000 7fece8937a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:13.696 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:13.695+0000 7fece8937a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:13.715 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:13.715 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:13.715 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.716 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:13.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:13.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:13.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:13.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:27:13.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:13.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.723 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:13.724 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:13.724 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:13.724 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:13.724 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:13Z" 2024-06-15T10:27:13.724 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:13.725 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:27:18.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:18.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:18.727 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:18.771 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:18.771 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:18.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:18.815+0000 7f32dea13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:18.818 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:18.815+0000 7f32dea13a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.840 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:18.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:18.842 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:18.843 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:18Z" 2024-06-15T10:27:18.845 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:18.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:27:23.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:23.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:23.847 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:23.898 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:23.899 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:23.954 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:23.951+0000 7f7a55f27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:23.955 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:23.955+0000 7f7a55f27a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:23Z" 2024-06-15T10:27:23.979 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:23.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:27:28.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:28.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:28.982 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:29.027 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:29.027 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:29.074 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:29.075+0000 7f78f6b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:29.075 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:29.075+0000 7f78f6b70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:29.099 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:29Z" 2024-06-15T10:27:29.099 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:29.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:27:34.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:34.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:34.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:34.144 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:34.144 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:34.190 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:34.191+0000 7f23019d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:34.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:34.191+0000 7f23019d8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:34.215 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:34.215 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:34.215 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:34.215 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:34.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.762966_670.1", 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.762966Z", 2024-06-15T10:27:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:34Z" 2024-06-15T10:27:34.220 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:34.221 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.762966_670.1 2024-06-15T10:27:39.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:39.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:39.222 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:39.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:39.266 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:39.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:39.327+0000 7f4e98447a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:39.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:39.327+0000 7f4e98447a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.352 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.353 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:39.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.762966_670.1", 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.762966Z", 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:39.355 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:39.356 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:39Z" 2024-06-15T10:27:39.357 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:39.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.762966_670.1 2024-06-15T10:27:44.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:44.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:44.359 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:44.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:44.403 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:44.447 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:44.447+0000 7f6bb700ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:44.448 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:44.447+0000 7f6bb700ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:44.474 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:44.475 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:44.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:44.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:44.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:44Z" 2024-06-15T10:27:44.479 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:44.480 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:27:49.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:49.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:49.481 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:49.525 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:49.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:49.581 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:49.579+0000 7f3db38bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:49.582 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:49.579+0000 7f3db38bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:49.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.605 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:49.607 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:49.608 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:49Z" 2024-06-15T10:27:49.609 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:49.610 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:27:54.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:54.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:54.611 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:54.654 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:54.654 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:54.696 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:54.695+0000 7f20e5641a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:54.697 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:54.695+0000 7f20e5641a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:54.716 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:54.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:54.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:54.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:54.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:54.721 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:54Z" 2024-06-15T10:27:54.722 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:54.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:27:59.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:59.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:27:59.724 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:27:59.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:27:59.767 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:27:59.812 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:59.811+0000 7f8b9d365a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:59.813 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:27:59.811+0000 7f8b9d365a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:27:59.832 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:27:59.832 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:27:59.832 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:27:59.833 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:59.834 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:27:59.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:59.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:27:59Z" 2024-06-15T10:27:59.837 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:27:59.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:28:04.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:04.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:04.839 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:04.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:04.883 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:04.924 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:04.922+0000 7f9f9a74ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:04.925 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:04.922+0000 7f9f9a74ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:04.944 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:04.944 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:04.944 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:04.945 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:04.946 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:04.947 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:04.948 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:04Z" 2024-06-15T10:28:04.949 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:04.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:28:09.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:09.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:09.951 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:09.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:09.994 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:10.038 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:10.038+0000 7f81f8330a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:10.040 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:10.038+0000 7f81f8330a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:10.058 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:10.058 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:10.058 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:10.059 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:10.060 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:10.061 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:10.062 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:10Z" 2024-06-15T10:28:10.063 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:10.064 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:28:15.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:15.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:15.065 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:15.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:15.108 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:15.178 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:15.174+0000 7fb58fce8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:15.179 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:15.178+0000 7fb58fce8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:15.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:15.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.211 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:15.212 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:15.213 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:15.214 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:15.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:15.214 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:15Z" 2024-06-15T10:28:15.214 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:15.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:28:20.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:20.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:20.216 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:20.260 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:20.260 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:20.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:20.318+0000 7f6e29ce4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:20.323 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:20.322+0000 7f6e29ce4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:20.348 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:20.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:20.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:20.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:20.353 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:20.354 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.355 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:20.356 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:20.356 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:20.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:20.356 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:20Z" 2024-06-15T10:28:20.356 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:20.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:28:25.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:25.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:25.358 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:25.407 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:25.407 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:25.451 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:25.450+0000 7f935da22a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:25.452 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:25.450+0000 7f935da22a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:25.475 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:25.476 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.477 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:25.478 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:25.479 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:25.480 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:25.480 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:25.480 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:25Z" 2024-06-15T10:28:25.480 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:25.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:28:30.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:30.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:30.482 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:30.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:30.526 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:30.585 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:30.582+0000 7f85e0cf2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:30.586 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:30.586+0000 7f85e0cf2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:30.613 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:30Z" 2024-06-15T10:28:30.618 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:30.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:28:35.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:35.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:35.620 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:35.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:35.662 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:35.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:35.714+0000 7fa10802ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:35.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:35.718+0000 7fa10802ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:35.744 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.745 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:35.746 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:35.747 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:35.748 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:35.749 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:35.750 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:35.750 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:35.750 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:35Z" 2024-06-15T10:28:35.750 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:35.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:28:40.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:40.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:40.752 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:40.796 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:40.797 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:40.850 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:40.846+0000 7fd8ff166a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:40.851 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:40.850+0000 7fd8ff166a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:40.874 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:40.875 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:40.876 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:40.877 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:40.878 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:40.879 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:40Z" 2024-06-15T10:28:40.880 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:40.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:28:45.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:45.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:45.882 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:45.925 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:45.925 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:45.977 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:45.978+0000 7f180956aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:45.978 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:45.978+0000 7f180956aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:45.996 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:45.996 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:45.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:45.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:45.999 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:46.000 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:45Z" 2024-06-15T10:28:46.001 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:46.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:28:51.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:51.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:51.004 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:51.045 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:51.045 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:51.093 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:51.094+0000 7fb2366b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:51.094 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:51.098+0000 7fb2366b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:51.112 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:51.113 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:51.114 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:51.115 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:51.116 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:51Z" 2024-06-15T10:28:51.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:51.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:28:56.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:56.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:28:56.120 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:28:56.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:28:56.164 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:28:56.208 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:56.209+0000 7f099b613a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:56.209 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:28:56.209+0000 7f099b613a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:28:56.238 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:28:56.239 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:56.240 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:56.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:28:56.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:28:56Z" 2024-06-15T10:28:56.243 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:28:56.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:01.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:01.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:01.246 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:01.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:01.289 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:01.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:01.333+0000 7f9d7f793a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:01.333 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:01.333+0000 7f9d7f793a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:01.358 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:01.358 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:01.359 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:01.361 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:01.362 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:01Z" 2024-06-15T10:29:01.363 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:01.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:06.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:06.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:06.365 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:06.420 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:06.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:06.481+0000 7f2e40d01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:06.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:06.481+0000 7f2e40d01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:06.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:06.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:06.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:06.511 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:06Z" 2024-06-15T10:29:06.511 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:06.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:11.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:11.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:11.513 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:11.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:11.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:11.609 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:11.609+0000 7f49b2058a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:11.610 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:11.609+0000 7f49b2058a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:11.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:11.630 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:11.631 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.270971_594.1", 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.270971Z", 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:11.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:11Z" 2024-06-15T10:29:11.633 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:11.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.270971_594.1 2024-06-15T10:29:16.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:16.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:16.636 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:16.679 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:16.679 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:16.740 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:16.741+0000 7f26a9e2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:16.741 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:16.741+0000 7f26a9e2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:16.767 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:16.767 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:16.767 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:16.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:16.769 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:16.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:16.771 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:16.772 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:16Z" 2024-06-15T10:29:16.773 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:16.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:21.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:21.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:21.775 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:21.818 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:21.819 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:21.865 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:21.865+0000 7fed10ab8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:21.866 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:21.869+0000 7fed10ab8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:21.889 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:21.890 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:21.891 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:21.892 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:21Z" 2024-06-15T10:29:21.893 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:21.894 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:26.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:26.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:26.895 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:26.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:26.939 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:26.990 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:26.993+0000 7f3c9ba1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:26.991 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:26.993+0000 7f3c9ba1ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.011 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:27.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:27.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:27.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:26Z" 2024-06-15T10:29:27.016 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:27.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:29:32.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:32.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:32.018 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:32.074 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:32.074 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:32.119 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:32.121+0000 7fe0cebc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:32.120 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:32.121+0000 7fe0cebc9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:32.137 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:32.138 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:32.139 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:32.140 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:32.141 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:32.142 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:32.143 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:32Z" 2024-06-15T10:29:32.143 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:32.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:37.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:37.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:37.145 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:37.188 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:37.188 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:37.237 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:37.237+0000 7fd09fceaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:37.238 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:37.241+0000 7fd09fceaa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:37.264 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:37.265 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:37.266 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:37.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:37.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:37.269 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:37.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:37.270 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:37Z" 2024-06-15T10:29:37.270 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:37.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:29:42.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:42.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:42.272 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:42.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:42.317 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:42.371 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:42.373+0000 7f8820117a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:42.372 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:42.373+0000 7f8820117a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:42.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:42.393 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:42.394 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:42.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:42.396 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:42.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:42.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:42.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:42Z" 2024-06-15T10:29:42.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:42.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:29:47.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:47.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:47.403 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:47.446 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:47.446 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:47.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:47.508+0000 7f299aba5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:47.509 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:47.508+0000 7f299aba5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.528 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:47.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:47.532 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:47.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:47.533 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:47Z" 2024-06-15T10:29:47.533 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:47.534 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:29:52.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:52.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:52.535 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:52.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:52.580 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:52.629 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:52.628+0000 7f4e05716a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:52.630 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:52.628+0000 7f4e05716a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:52.650 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:52.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:52.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:52.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:52.654 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:52Z" 2024-06-15T10:29:52.655 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:52.656 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:29:57.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:57.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:29:57.658 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:29:57.702 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:29:57.702 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:29:57.754 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:57.756+0000 7fc9c6ea7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:57.755 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:29:57.756+0000 7fc9c6ea7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:29:57.773 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:57.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:29:57.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:57.776 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:29:57.777 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:29:57.778 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:29:57Z" 2024-06-15T10:29:57.779 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:29:57.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:30:02.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:02.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:02.782 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:02.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:02.826 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:02.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:02.884+0000 7fd1e1a12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:02.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:02.884+0000 7fd1e1a12a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:02.903 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:02.904 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:02.905 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.906 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:02.907 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:02Z" 2024-06-15T10:30:02.908 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:02.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:30:07.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:07.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:07.910 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:07.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:07.968 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:08.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:08.020+0000 7fc5d4c1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:08.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:08.020+0000 7fc5d4c1da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:08.042 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:08.042 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:08.042 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:08.042 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:08.042 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:08.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:08.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:30:08.045 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:08.046 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:08Z" 2024-06-15T10:30:08.047 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:08.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:30:13.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:13.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:13.050 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:13.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:13.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:13.152 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:13.152+0000 7f0725f6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:13.153 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:13.152+0000 7f0725f6ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:13.178 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:13.179 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:13.180 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:13.181 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:13.182 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:13Z" 2024-06-15T10:30:13.183 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:13.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:30:18.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:18.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:18.185 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:18.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:18.229 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:18.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:18.280+0000 7ff83f93da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:18.281 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:18.280+0000 7ff83f93da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:18.310 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:18.312 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:18.313 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:18.314 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:18.315 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:18Z" 2024-06-15T10:30:18.315 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:18.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:30:23.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:23.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:23.317 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:23.379 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:23.379 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:23.436 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:23.436+0000 7fbf4589ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:23.437 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:23.436+0000 7fbf4589ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:23.455 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:23.456 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:23.457 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:23.458 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.459 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:23.460 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:23.460 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:23.460 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:23.460 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:23Z" 2024-06-15T10:30:23.460 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:23.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:30:28.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:28.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:28.462 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:28.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:28.505 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:28.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:28.552+0000 7f945573ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:28.555 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:28.556+0000 7f945573ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:28.582 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:28.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:28.584 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:30:28.585 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:28.586 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:28Z" 2024-06-15T10:30:28.587 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:28.588 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:30:33.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:33.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:33.590 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:33.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:33.632 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:33.695 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:33.696+0000 7f20fc007a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:33.696 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:33.696+0000 7f20fc007a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:33.716 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:33.717 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:33.718 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:33.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:33.721 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:33.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:33.721 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:33Z" 2024-06-15T10:30:33.721 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:33.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:30:38.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:38.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:38.723 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:38.766 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:38.766 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:38.815 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:38.815+0000 7f8c8cd75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:38.816 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:38.815+0000 7f8c8cd75a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:38.834 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:38.835 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:38.836 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:38.837 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:38.838 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:38Z" 2024-06-15T10:30:38.839 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:38.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:30:43.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:43.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:43.842 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:43.887 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:43.887 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:43.936 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:43.935+0000 7ff741b6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:43.937 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:43.935+0000 7ff741b6aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:43.959 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:43.959 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:43.960 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:43.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:43.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:43.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:43.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:43.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:43.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:43.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:43.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:43.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:43.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:43.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:43.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:43.970 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:43.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:43.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.970 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:43.970 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:43.971 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:43.972 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:43.972 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:43.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:43.972 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:43.972 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:43.973 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:43Z" 2024-06-15T10:30:43.974 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:43.976 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:30:48.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:48.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:48.978 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:49.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:49.035 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:49.121 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:49.119+0000 7f724d90ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:49.122 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:49.119+0000 7f724d90ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:49.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:49.148 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:49.148 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:49.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:49.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:49.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:49.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:49.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:49.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:49.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:49.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:49.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:49.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:49.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:49.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:49.158 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:49Z" 2024-06-15T10:30:49.158 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:49.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:30:54.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:54.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:54.160 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:54.203 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:54.204 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:54.247 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:54.247+0000 7f348d546a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:54.247 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:54.247+0000 7f348d546a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:54.266 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:54.267 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:54.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:54.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:54.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:54.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:54.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:54.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:54.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:54.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:54.273 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:54Z" 2024-06-15T10:30:54.274 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:54.275 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:30:59.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:59.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:30:59.276 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:30:59.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:30:59.319 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:30:59.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:59.363+0000 7ffa94b2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:59.367 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:30:59.367+0000 7ffa94b2fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:30:59.388 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:30:59.388 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:59.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:59.390 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:59.391 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:30:59.392 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:30:59Z" 2024-06-15T10:30:59.393 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:30:59.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:31:04.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:04.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:04.395 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:04.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:04.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:04.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:04.483+0000 7f5033a09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:04.486 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:04.487+0000 7f5033a09a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:04.506 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:04.507 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:31:04.508 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:04.509 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:04Z" 2024-06-15T10:31:04.510 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:04.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:31:09.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:09.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:09.513 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:09.556 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:09.557 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:09.599 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:09.599+0000 7fa08a2f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:09.600 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:09.599+0000 7fa08a2f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:09.618 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:09.619 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.620 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.621 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:09.622 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:09.623 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:09Z" 2024-06-15T10:31:09.624 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:09.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:31:14.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:14.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:14.627 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:14.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:14.676 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:14.718 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:14.715+0000 7f251fe7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:14.719 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:14.719+0000 7f251fe7da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:14.738 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:14.739 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:14.740 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:14.741 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:14.742 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:14.743 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:14.744 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:14.744 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:14Z" 2024-06-15T10:31:14.744 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:14.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:31:19.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:19.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:19.746 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:19.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:19.789 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:19.839 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:19.839+0000 7feb294d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:19.840 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:19.839+0000 7feb294d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:19.861 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.862 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:19.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:19.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:19.866 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:19.867 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:19.868 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.869 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:19.869 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:19.869 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:19.869 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:19.869 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:19Z" 2024-06-15T10:31:19.870 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:19.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:31:24.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:24.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:24.872 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:24.917 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:24.917 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:24.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:24.970+0000 7f36e2504a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:24.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:24.974+0000 7f36e2504a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:25.001 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:25.001 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:25.002 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:25.003 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:25.004 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:25.005 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:24Z" 2024-06-15T10:31:25.006 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:25.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:31:30.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:30.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:30.008 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:30.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:30.063 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:30.127 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:30.126+0000 7f9063372a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:30.128 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:30.126+0000 7f9063372a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:30.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:30.149 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:30.149 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:30.149 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:30.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:30.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:30.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:30.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:30.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.152 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:30.152 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:30.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:30.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:30.154 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:30Z" 2024-06-15T10:31:30.155 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:30.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:31:35.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:35.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:35.158 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:35.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:35.202 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:35.245 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:35.242+0000 7f027f4f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:35.246 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:35.246+0000 7f027f4f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:35.267 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:35.268 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.269 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:35.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:35.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:35.272 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:35Z" 2024-06-15T10:31:35.272 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:35.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:31:40.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:40.275 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:40.320 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:40.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:40.362+0000 7f0e1bb2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:40.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:40.362+0000 7f0e1bb2ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:40.382 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:40.382 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:40.382 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:40.382 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:40.383 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:40.384 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:40.385 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:40.386 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:40Z" 2024-06-15T10:31:40.387 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:40.388 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:31:45.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:45.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:45.391 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:45.435 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:45.435 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:45.478 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:45.474+0000 7f31da1bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:45.479 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:45.478+0000 7f31da1bba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:45.500 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:45.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:45.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:45.503 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:45.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:45Z" 2024-06-15T10:31:45.505 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:45.506 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:31:50.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:50.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:50.508 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:50.551 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:50.552 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:50.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:50.606+0000 7fef1bca8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:50.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:50.606+0000 7fef1bca8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:50.628 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:50.628 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:50.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:50.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:50.631 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:50.632 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:50Z" 2024-06-15T10:31:50.633 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:50.635 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:31:55.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:55.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:31:55.636 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:31:55.679 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:31:55.680 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:31:55.733 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:55.730+0000 7f83637a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:55.734 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:31:55.734+0000 7f83637a6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.753 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:55.754 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.755 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:31:55.756 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:31:55.757 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:31:55.758 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:31:55.758 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:31:55.758 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:31:55Z" 2024-06-15T10:31:55.758 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:31:55.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:32:00.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:00.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:00.760 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:00.803 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:00.804 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:00.882 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:00.882+0000 7f0d7796aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:00.883 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:00.882+0000 7f0d7796aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:00.908 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:00.908 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:00.909 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:00.910 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:00.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:00.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:00Z" 2024-06-15T10:32:00.913 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:00.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:32:05.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:05.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:05.915 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:05.959 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:05.960 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:06.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:06.014+0000 7f5171677a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:06.014 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:06.014+0000 7f5171677a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:06.040 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:06.040 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:06.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:06.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:06.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:06.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.769917_671.1", 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:06.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.769917Z", 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:06.044 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:06Z" 2024-06-15T10:32:06.045 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:06.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.769917_671.1 2024-06-15T10:32:11.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:11.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:11.049 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:11.092 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:11.093 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:11.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:11.146+0000 7f4192748a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:11.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:11.146+0000 7f4192748a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:11.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:11.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:11.166 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:11.168 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:11Z" 2024-06-15T10:32:11.169 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:11.170 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:32:16.171 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... FAIL 2024-06-15T10:32:16.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:16.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:16.172 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:16.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:16.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:16.275 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:32:16.299 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:32:16.299 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:32:16.299 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:32:16.300 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:16.301 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:32:16.301 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:32:16.301 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:32:16.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:32:16.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:16.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:16.302 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:16.346 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:16.346 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:16.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:16.425+0000 7f1c67992a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:16.426 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:16.429+0000 7f1c67992a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:16.446 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:16.446 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:16.446 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:16.446 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:16.446 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:16.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:16.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:16.449 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:16.450 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:16Z" 2024-06-15T10:32:16.451 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:16.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:32:21.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:21.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:21.454 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:21.498 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:21.498 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:21.547 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:21.549+0000 7f65edcbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:21.548 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:21.549+0000 7f65edcbea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:21.567 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:21.567 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:21.567 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:21.567 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:21.568 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:21.569 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:32:21.570 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:21.571 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:21Z" 2024-06-15T10:32:21.572 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:21.573 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:32:26.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:26.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:26.575 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:26.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:26.620 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:26.675 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:26.677+0000 7f0e434a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:26.676 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:26.677+0000 7f0e434a7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:26.697 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:26.697 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:26.698 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:26.699 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:32:26.700 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:26.701 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:26Z" 2024-06-15T10:32:26.702 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:26.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:32:31.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:31.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:31.705 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:31.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:31.756 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:31.824 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:31.825+0000 7fba5a790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:31.825 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:31.825+0000 7fba5a790a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:31.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:31.850 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.851 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:31.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:31.853 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:31.854 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:31.854 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:31.854 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:31Z" 2024-06-15T10:32:31.854 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:31.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:32:36.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:36.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:36.855 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:36.899 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:36.899 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:36.940 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:36.941+0000 7fb56e438a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:36.941 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:36.941+0000 7fb56e438a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:36.960 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:36.960 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:36.960 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:36.960 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:36.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:36.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:36.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:36.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:36Z" 2024-06-15T10:32:36.965 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:36.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:32:41.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:41.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:41.968 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:42.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:42.011 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:42.061 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:42.061+0000 7f40fc60ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:42.062 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:42.061+0000 7f40fc60ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:42.081 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:42.082 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:42.083 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:42.084 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:42.085 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:42Z" 2024-06-15T10:32:42.086 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:42.087 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:32:47.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:47.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:47.088 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:47.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:47.132 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:47.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:47.261+0000 7f98f6021a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:47.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:47.261+0000 7f98f6021a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:47.284 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:47.285 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:32:47.286 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:47.287 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:47.288 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:47Z" 2024-06-15T10:32:47.289 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:47.290 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:32:52.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:52.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:52.291 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:52.334 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:52.335 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:52.398 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:52.401+0000 7f35ddc66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:52.399 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:52.401+0000 7f35ddc66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:52.424 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:52.424 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:52.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:52.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:52.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:52.428 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:52.429 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:52Z" 2024-06-15T10:32:52.430 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:52.431 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:32:57.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:57.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:32:57.432 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:32:57.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:32:57.475 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:32:57.516 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:57.517+0000 7fb7400c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:57.517 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:32:57.517+0000 7fb7400c8a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:32:57.535 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:57.536 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:32:57.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:32:57.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:32:57.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:32:57Z" 2024-06-15T10:32:57.540 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:32:57.541 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:33:02.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:02.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:02.543 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:02.585 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:02.585 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:02.641 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:02.641+0000 7fb108909a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:02.642 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:02.645+0000 7fb108909a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:02.668 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:02.668 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:02.668 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:02.668 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:02.669 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:02.670 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:02.671 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:02.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:02.674 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:02Z" 2024-06-15T10:33:02.674 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:02.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:33:07.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:07.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:07.676 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:07.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:07.720 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:07.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:07.760+0000 7f02d5e01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:07.763 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:07.764+0000 7f02d5e01a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:07.781 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:07.781 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:07.781 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:07.782 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:07.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:07.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:07.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:07Z" 2024-06-15T10:33:07.786 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:07.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:33:12.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:12.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:12.789 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:12.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:12.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:12.877 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:12.876+0000 7f7ffad03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:12.878 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:12.876+0000 7f7ffad03a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.897 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:12.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:12.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:12.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:12Z" 2024-06-15T10:33:12.902 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:12.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:33:17.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:17.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:17.905 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:17.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:17.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:18.000 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:18.000+0000 7f37fb774a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:18.001 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:18.000+0000 7f37fb774a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:18.021 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:18.022 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:18.023 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:18.024 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:18.025 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:18.026 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:18.027 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:18.027 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:18Z" 2024-06-15T10:33:18.027 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:18.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:33:23.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:23.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:23.030 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:23.077 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:23.077 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:23.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:23.132+0000 7ff1da0a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:23.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:23.136+0000 7ff1da0a3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:23.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:23.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:23.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:23.159 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:23.160 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:23.161 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:23Z" 2024-06-15T10:33:23.161 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:23.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:33:28.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:28.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:28.163 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:28.211 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:28.212 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:28.279 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:28.280+0000 7fe55f233a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:28.280 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:28.280+0000 7fe55f233a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.306 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:28.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:28.308 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:28.309 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:28.310 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:28.311 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:28Z" 2024-06-15T10:33:28.312 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:28.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:33:33.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:33.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:33.315 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:33.359 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:33.360 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:33.401 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:33.400+0000 7f56441dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:33.402 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:33.400+0000 7f56441dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:33.422 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:33.422 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:33.422 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:33.422 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:33.423 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:33.424 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:33.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:33.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:33Z" 2024-06-15T10:33:33.427 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:33.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:33:38.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:38.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:38.429 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:38.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:38.475 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:38.518 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:38.516+0000 7f01228b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:38.519 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:38.520+0000 7f01228b1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:38.537 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:38.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:38.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:38.540 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:38.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:38Z" 2024-06-15T10:33:38.542 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:38.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:33:43.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:43.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:43.544 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:43.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:43.592 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:43.640 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:43.640+0000 7f34da068a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:43.641 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:43.640+0000 7f34da068a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:43.670 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:43.671 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:43.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:43.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:43.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:43Z" 2024-06-15T10:33:43.675 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:43.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:33:48.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:48.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:48.677 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:48.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:48.721 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:48.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:48.764+0000 7f7edefa1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:48.764 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:48.764+0000 7f7edefa1a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:48.786 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:48.786 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:48.786 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:48.787 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:48.788 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:48.789 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:48.790 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:48Z" 2024-06-15T10:33:48.791 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:48.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:33:53.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:53.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:53.792 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:53.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:53.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:53.888 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:53.888+0000 7f05dea69a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:53.889 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:53.888+0000 7f05dea69a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:53.913 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:53.913 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:53.913 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:53.913 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:53.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:53.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:53.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:53.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:53Z" 2024-06-15T10:33:53.919 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:53.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:33:58.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:58.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:33:58.922 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:33:58.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:33:58.966 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:33:59.015 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:59.015+0000 7f5671a7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:59.016 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:33:59.015+0000 7f5671a7aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:33:59.037 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:59.038 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.039 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:33:59.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:59.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:33:59Z" 2024-06-15T10:33:59.042 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:33:59.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:34:04.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:04.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:04.045 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:04.088 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:04.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:04.138 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:04.139+0000 7f03c8115a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:04.140 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:04.139+0000 7f03c8115a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:04.161 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:04.161 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:04.161 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:04.162 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:04.163 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:04.164 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:04.165 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:04.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:04.167 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:04Z" 2024-06-15T10:34:04.167 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:04.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:34:09.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:09.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:09.169 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:09.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:09.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:09.259 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:09.259+0000 7fcbf75d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:09.260 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:09.259+0000 7fcbf75d0a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:09.288 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:09.289 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.290 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:09.291 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.817128_675.1", 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.817128Z", 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:09.292 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:09.293 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:09.294 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:09Z" 2024-06-15T10:34:09.294 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:09.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.817128_675.1 2024-06-15T10:34:14.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:14.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:14.297 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:14.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:14.341 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:14.393 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:14.391+0000 7f538d5ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:14.394 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:14.391+0000 7f538d5ada40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:14.413 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:14.414 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:14.415 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:14.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:14Z" 2024-06-15T10:34:14.418 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:14.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:34:19.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:19.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:19.420 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:19.464 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:19.464 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:19.512 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:19.511+0000 7f560880da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:19.513 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:19.511+0000 7f560880da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:19.532 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:19.533 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:19.534 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:34:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:19.536 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:19.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:19.538 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:19Z" 2024-06-15T10:34:19.538 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:19.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:34:24.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:24.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:24.540 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:24.600 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:24.601 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:24.671 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:24.671+0000 7fbe202a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:24.672 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:24.671+0000 7fbe202a4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:24.693 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:24.693 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:24.693 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:24.693 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:24.693 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:24.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:24.695 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:24.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:24Z" 2024-06-15T10:34:24.698 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:24.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:34:29.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:29.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:29.700 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:29.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:29.745 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:29.793 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:29.791+0000 7f4cd0ac6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:29.794 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:29.795+0000 7f4cd0ac6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:29.816 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:29.817 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.818 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:29.819 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.820 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:29.821 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:29.821 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:29.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:29.821 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:29Z" 2024-06-15T10:34:29.821 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:29.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:34:34.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:34.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:34.823 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:34.867 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:34.867 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:34.930 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:34.931+0000 7fc115a94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:34.931 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:34.931+0000 7fc115a94a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.958 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:34.959 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:34.960 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:34.961 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:34.962 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:34Z" 2024-06-15T10:34:34.963 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:34.964 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:34:39.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:39.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:39.965 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:40.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:40.009 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:40.069 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:40.067+0000 7f40596f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:40.070 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:40.067+0000 7f40596f5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:40.089 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:40.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:40.091 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:40.092 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:40.093 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:40Z" 2024-06-15T10:34:40.094 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:40.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:34:45.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:45.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:45.097 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:45.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:45.141 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:45.201 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:45.199+0000 7fd8a33b6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:45.202 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:45.199+0000 7fd8a33b6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:45.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:45.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:45.225 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:45.227 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:45.228 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:45.228 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:45.228 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:45Z" 2024-06-15T10:34:45.228 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:45.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:34:50.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:50.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:50.230 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:50.278 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:50.278 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:50.339 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:50.338+0000 7fe41905aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:50.340 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:50.338+0000 7fe41905aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:50.366 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:50.367 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:50.368 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:50.369 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:50.370 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:50Z" 2024-06-15T10:34:50.371 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:50.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:34:55.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:55.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:34:55.374 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:34:55.417 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:34:55.417 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:34:55.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:55.466+0000 7f018ff17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:55.468 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:34:55.466+0000 7f018ff17a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:34:55.490 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:34:55.490 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:34:55.490 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:34:55.490 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:34:55.491 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:55.492 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:34:55.494 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:34:55Z" 2024-06-15T10:34:55.495 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:34:55.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:35:00.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:00.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:00.497 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:00.544 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:00.545 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:00.598 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:00.598+0000 7fc187d79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:00.599 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:00.598+0000 7fc187d79a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:00.625 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:00.625 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:00.625 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:00.625 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:00.626 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:00.627 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:00.628 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:00.629 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:00Z" 2024-06-15T10:35:00.630 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:00.631 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:05.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:05.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:05.633 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:05.686 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:05.687 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:05.761 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:05.758+0000 7f066853da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:05.762 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:05.758+0000 7f066853da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:05.782 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:05.782 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:05.783 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:05.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:05.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:05.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:05Z" 2024-06-15T10:35:05.787 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:05.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:35:10.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:10.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:10.789 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:10.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:10.833 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:10.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:10.886+0000 7f52e09efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:10.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:10.890+0000 7f52e09efa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:10.910 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:10.911 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:10.912 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.913 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:10.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:10Z" 2024-06-15T10:35:10.915 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:10.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:15.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:15.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:15.917 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:15.959 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:15.959 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:16.010 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:16.006+0000 7f80835c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:16.011 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:16.010+0000 7f80835c9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:16.030 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:16.031 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:16.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:16.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:16.034 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:16Z" 2024-06-15T10:35:16.035 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:16.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:21.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:21.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:21.037 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:21.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:21.080 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:21.133 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:21.130+0000 7fc46ce77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:21.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:21.130+0000 7fc46ce77a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:21.153 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:21.153 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:21.153 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:21.153 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:21.153 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:21.154 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:21.155 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:21.156 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:21.157 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:21Z" 2024-06-15T10:35:21.158 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:21.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:26.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:26.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:26.161 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:26.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:26.205 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:26.248 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:26.246+0000 7fe7bcf48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:26.249 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:26.246+0000 7fe7bcf48a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:26.268 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:26.268 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:26.268 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:26.269 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:26.269 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:26.269 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:26.269 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:26.269 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.270 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:26.270 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:26.270 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:26.270 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:26.270 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:26.271 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:26.272 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:26.273 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:26.274 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:26Z" 2024-06-15T10:35:26.275 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:26.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:35:31.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:31.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:31.278 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:31.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:31.326 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:31.395 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:31.398+0000 7f458a324a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:31.395 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:31.398+0000 7f458a324a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:31.415 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:31.415 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:31.415 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:31.415 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:31.415 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:31.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:31.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:31.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:31.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:31Z" 2024-06-15T10:35:31.420 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:31.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:35:36.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:36.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:36.422 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:36.465 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:36.465 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:36.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:36.522+0000 7ff32f719a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:36.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:36.522+0000 7ff32f719a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:36.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:36.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:36.549 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:36.550 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:36Z" 2024-06-15T10:35:36.552 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:36.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:41.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:41.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:41.554 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:41.599 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:41.599 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:41.650 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:41.649+0000 7fa10dacda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:41.651 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:41.653+0000 7fa10dacda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:41.671 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:41.671 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:41.671 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:41.671 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:41.671 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:41.672 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:41.673 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:41.674 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:41.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:41Z" 2024-06-15T10:35:41.676 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:41.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:35:46.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:46.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:46.683 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:46.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:46.726 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:46.777 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:46.777+0000 7fcb47331a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:46.777 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:46.777+0000 7fcb47331a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:46.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:46.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:46.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:46.802 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.803 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:46.804 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:46.804 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:46.804 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:46.804 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:46Z" 2024-06-15T10:35:46.804 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:46.805 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:35:51.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:51.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:51.807 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:51.852 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:51.852 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:51.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:51.893+0000 7f3abf6faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:51.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:51.897+0000 7f3abf6faa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:51.913 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:51.913 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:51.914 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:51.914 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:51.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:51.914 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:51.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:51.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.915 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:51.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:51.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:51.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:51.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:35:51.920 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:51.921 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:51.922 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:51.923 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:51Z" 2024-06-15T10:35:51.924 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:51.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:35:56.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:56.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:35:56.927 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:35:56.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:35:56.971 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:35:57.012 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:57.013+0000 7f76ff228a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:57.013 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:35:57.013+0000 7f76ff228a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:57.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:57.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:35:57.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:35:57.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:35:57.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:35:57.037 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:35:57Z" 2024-06-15T10:35:57.037 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:35:57.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:36:02.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:02.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:02.039 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:02.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:02.083 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:02.145 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:02.145+0000 7f478a87fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:02.145 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:02.145+0000 7f478a87fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:02.171 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:02.171 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:02.171 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:02.172 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:02.173 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:02.174 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:02.175 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:02Z" 2024-06-15T10:36:02.176 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:02.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:36:07.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:07.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:07.179 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:07.223 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:07.223 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:07.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:07.277+0000 7f05fc530a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:07.278 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:07.281+0000 7f05fc530a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:07.303 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:07.304 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:07.305 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:07.306 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:07Z" 2024-06-15T10:36:07.307 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:07.308 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:36:12.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:12.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:12.310 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:12.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:12.353 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:12.403 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:12.405+0000 7f5a150f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:12.404 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:12.405+0000 7f5a150f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:12.422 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:12.423 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:12.424 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:12.424 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:12.424 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:12.424 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:12.425 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:36:12.426 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:12.427 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:12Z" 2024-06-15T10:36:12.428 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:12.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:36:17.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:17.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:17.431 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:17.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:17.474 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:17.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:17.521+0000 7f4658634a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:17.522 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:17.525+0000 7f4658634a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.541 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:17.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:17.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:17.545 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:17.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:17.547 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:17Z" 2024-06-15T10:36:17.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:17.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:36:22.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:22.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:22.549 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:22.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:22.593 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:22.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:22.645+0000 7f07bf123a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:22.647 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:22.649+0000 7f07bf123a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:22.673 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:22.673 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:22.673 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:22.673 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:22.675 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:22.676 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:22.677 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:22Z" 2024-06-15T10:36:22.678 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:22.679 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:36:27.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:27.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:27.680 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:27.724 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:27.724 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:27.776 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:27.777+0000 7f776a9eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:27.777 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:27.777+0000 7f776a9eca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:27.797 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:27.798 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:27.799 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.800 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.801 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:27.802 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:27.802 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:27.802 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:27.802 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:27Z" 2024-06-15T10:36:27.802 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:27.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:36:32.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:32.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:32.804 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:32.848 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:32.848 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:32.894 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:32.892+0000 7f12256eba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:32.895 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:32.896+0000 7f12256eba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:32.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:32.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:32.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:32.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:32.919 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:32.919 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:32.919 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:32Z" 2024-06-15T10:36:32.919 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:32.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:36:37.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:37.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:37.921 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:37.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:37.965 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:38.021 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:38.020+0000 7f2cf44b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:38.022 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:38.020+0000 7f2cf44b5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:38.046 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:38.047 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.048 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:38.049 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:38.050 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:38.051 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:38.052 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:38.052 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:38Z" 2024-06-15T10:36:38.052 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:38.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:36:43.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:43.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:43.054 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:43.097 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:43.098 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:43.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:43.188+0000 7f9b6e05ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:43.187 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:43.188+0000 7f9b6e05ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:43.206 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:43.206 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:43.207 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:43.208 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:36:43.209 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:43.210 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:43Z" 2024-06-15T10:36:43.211 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:43.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:36:48.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:48.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:48.213 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:48.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:48.256 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:48.306 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:48.304+0000 7f8c25a78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:48.307 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:48.308+0000 7f8c25a78a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:48.327 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:48.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:48.329 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:48.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:48.331 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:48Z" 2024-06-15T10:36:48.332 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:48.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:36:53.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:53.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:53.335 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:53.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:53.378 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:53.424 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:53.424+0000 7fecbb4fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:53.425 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:53.424+0000 7fecbb4fda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:53.444 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:53.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:53.446 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:53.447 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:53.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:53.449 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:53Z" 2024-06-15T10:36:53.449 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:53.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:36:58.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:58.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:36:58.452 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:36:58.494 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:36:58.494 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:36:58.557 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:58.556+0000 7f7ecf81ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:58.557 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:36:58.556+0000 7f7ecf81ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:58.579 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:58.580 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:36:58.581 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:58.582 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:36:58.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:36:58Z" 2024-06-15T10:36:58.584 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:36:58.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:37:03.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:03.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:03.586 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:03.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:03.629 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:03.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:03.680+0000 7fbe1555aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:03.682 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:03.680+0000 7fbe1555aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:03.709 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:03.710 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:03.711 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:03.712 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:03.713 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:03Z" 2024-06-15T10:37:03.714 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:03.716 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:37:08.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:08.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:08.718 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:08.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:08.762 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:08.816 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:08.816+0000 7fd46c674a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:08.817 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:08.816+0000 7fd46c674a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:08.838 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:08.839 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.840 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:08.841 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:08.842 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:08.843 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:08.843 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:08.843 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:08Z" 2024-06-15T10:37:08.843 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:08.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:37:13.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:13.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:13.846 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:13.903 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:13.904 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:13.945 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:13.944+0000 7f2c1a886a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:13.946 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:13.948+0000 7f2c1a886a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:13.965 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:13.965 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:13.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:13.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:37:13.968 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:13.969 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:13Z" 2024-06-15T10:37:13.970 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:13.971 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:37:18.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:18.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:18.972 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:19.015 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:19.016 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:19.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:19.076+0000 7f12a4de6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:19.078 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:19.080+0000 7f12a4de6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:19.099 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:19.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:19.099 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:19.099 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:19.099 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:19.100 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:19.101 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:19.102 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:19.103 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:19.104 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:19.105 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:19.105 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:19.105 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:19Z" 2024-06-15T10:37:19.105 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:19.106 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:37:24.107 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-06-15T10:37:24.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:24.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:24.108 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:24.152 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:24.153 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:24.198 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=6634f146-140c-4941-81ef-ff44e7855a30 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T07:38:10.276409Z" 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:37:24.216 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.930258Z" 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718445381.083679_1521.1", 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:56:21.083679Z" 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: "last_update": "2024-06-15T09:30:38.978738Z" 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-06-15T10:37:24.217 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-06-15T10:37:24.218 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2024-06-15T10:37:24.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:24.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:24.219 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:24.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:24.262 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:24.306 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:24.307+0000 7fc0e593ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:24.307 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:24.307+0000 7fc0e593ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:24.326 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:24.327 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:24.328 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:24.329 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.330 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:24.331 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:24.331 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:24.331 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:24.331 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:24Z" 2024-06-15T10:37:24.331 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:24.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:37:29.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:29.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:29.333 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:29.392 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:29.392 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:29.520 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:29.519+0000 7fd5c7f42a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:29.521 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:29.519+0000 7fd5c7f42a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:29.541 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718440136.709769_896.1", 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:28:56.709769Z", 2024-06-15T10:37:29.548 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:29.549 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:29.549 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.549 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:29.549 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:29.550 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:29.550 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:29.550 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:29.550 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:29.552 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:29.553 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:29Z" 2024-06-15T10:37:29.553 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:29.554 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718440136.709769_896.1 2024-06-15T10:37:34.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:34.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:34.556 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:34.599 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:34.600 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:34.645 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:34.643+0000 7fb278590a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:34.646 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:34.643+0000 7fb278590a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.664 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:34.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:34.666 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:34.667 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:34.668 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:34Z" 2024-06-15T10:37:34.669 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:34.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:37:39.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:39.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:39.671 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:39.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:39.715 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:39.769 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:39.767+0000 7fd613520a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:39.770 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:39.767+0000 7fd613520a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:39.789 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:39.789 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:39.789 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:39.789 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:39.790 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:39.791 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:39.792 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:39.793 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:39Z" 2024-06-15T10:37:39.794 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:39.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:37:44.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:44.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:44.797 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:44.839 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:44.840 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:44.899 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:44.899+0000 7fbc35d70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:44.900 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:44.899+0000 7fbc35d70a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:44.924 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:44.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.926 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:44.927 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.928 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:44.929 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:44.929 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:44.929 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:44.929 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:44Z" 2024-06-15T10:37:44.929 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:44.930 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:37:49.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:49.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:49.931 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:49.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:49.976 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:50.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:50.019+0000 7f792ace6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:50.020 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:50.019+0000 7f792ace6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:50.039 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:50.039 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:50.040 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:50.041 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718439152.682869_749.1", 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:12:32.682869Z", 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:50Z" 2024-06-15T10:37:50.044 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:50.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718439152.682869_749.1 2024-06-15T10:37:55.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:55.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:37:55.046 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:37:55.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:37:55.089 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:37:55.154 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:55.151+0000 7ff7d6332a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:55.155 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:37:55.155+0000 7ff7d6332a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:37:55.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:55.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:55.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.223 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:37:55.224 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:37:55Z" 2024-06-15T10:37:55.225 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:37:55.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:38:00.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:00.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:00.228 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:00.271 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:00.271 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:00.327 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:00.327+0000 7f5587d3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:00.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:00.327+0000 7f5587d3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.349 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:00.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:00.351 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:00.352 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:00.353 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:00.354 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:00Z" 2024-06-15T10:38:00.354 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:00.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:05.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:05.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:05.357 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:05.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:05.414 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:05.476 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:05.475+0000 7f01cb86fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:05.477 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:05.475+0000 7f01cb86fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:05.497 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:05.497 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:05.499 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:05.500 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:05.502 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.503 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:05.503 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:05.503 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:05.504 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:05.505 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:05.506 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:05Z" 2024-06-15T10:38:05.506 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:05.507 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:10.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:10.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:10.508 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:10.552 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:10.552 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:10.611 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:10.611+0000 7f4724137a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:10.612 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:10.611+0000 7f4724137a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:10.632 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:10.632 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:10.632 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:10.633 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:10.634 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:10.635 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:10.636 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:10Z" 2024-06-15T10:38:10.637 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:10.638 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:15.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:15.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:15.640 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:15.685 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:15.685 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:15.744 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:15.742+0000 7f5755f0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:15.745 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:15.742+0000 7f5755f0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:15.770 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:15.770 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:15.770 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:15.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:15.772 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:15.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:15.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:15Z" 2024-06-15T10:38:15.775 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:15.776 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:20.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:20.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:20.777 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:20.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:20.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:20.861 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:20.858+0000 7fd78722ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:20.862 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:20.858+0000 7fd78722ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:20.880 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:20.880 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:20.881 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:20.882 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:20.883 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:20.884 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:20Z" 2024-06-15T10:38:20.885 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:20.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:38:25.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:25.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:25.888 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:25.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:25.931 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:25.974 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:25.970+0000 7f0ea9f0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:25.975 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:25.974+0000 7f0ea9f0ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:25.993 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.994 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.995 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:25.996 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.997 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:25.998 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:25.998 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:25.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:25.998 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:25Z" 2024-06-15T10:38:25.998 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:25.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:38:31.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:31.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:31.001 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:31.046 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:31.047 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:31.091 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:31.090+0000 7f2a80ce9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:31.092 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:31.090+0000 7f2a80ce9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:31.119 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:31.120 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:31.121 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:31.122 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:31.123 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:31.124 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:31Z" 2024-06-15T10:38:31.124 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:31.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:38:36.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:36.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:36.126 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:36.171 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:36.171 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:36.219 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:36.218+0000 7fe7d342ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:36.220 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:36.218+0000 7fe7d342ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:36.240 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:36.240 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:36.240 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:36.240 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:36.240 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:36.241 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:36.242 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:38:36.243 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:36.244 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:36Z" 2024-06-15T10:38:36.245 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:36.246 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:38:41.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:41.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:41.248 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:41.292 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:41.293 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:41.348 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:41.346+0000 7f6afeddda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:41.349 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:41.346+0000 7f6afeddda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:41.373 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:41.374 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:41.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.376 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:41.377 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:41.378 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:41Z" 2024-06-15T10:38:41.379 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:41.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:46.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:46.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:46.381 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:46.431 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:46.431 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:46.496 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:46.494+0000 7f8ba7068a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:46.497 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:46.494+0000 7f8ba7068a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:46.516 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:46.516 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:46.516 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:46.517 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:46.518 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:38:46.519 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:46Z" 2024-06-15T10:38:46.521 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:46.522 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:38:51.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:51.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:51.524 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:51.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:51.568 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:51.619 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:51.618+0000 7fe1d09f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:51.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:51.618+0000 7fe1d09f4a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:51.647 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:51.647 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:51.647 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:51.647 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:51.648 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:51.649 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:38:51.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:51.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:51Z" 2024-06-15T10:38:51.652 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:51.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:38:56.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:56.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:38:56.655 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:38:56.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:38:56.700 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:38:56.746 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:56.750+0000 7f7a5103fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:56.747 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:38:56.750+0000 7f7a5103fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:38:56.767 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:38:56.767 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:38:56.767 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:38:56.767 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:38:56.768 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:38:56.769 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:38:56.770 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:38:56.771 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:56.772 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:38:56.773 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:38:56.774 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:38:56.775 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:38:56.776 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:38:56Z" 2024-06-15T10:38:56.776 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:38:56.777 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:39:01.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:01.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:01.778 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:01.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:01.820 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:01.871 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:01.873+0000 7f86dd5d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:01.872 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:01.873+0000 7f86dd5d5a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:01.892 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:01.893 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.894 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:01.895 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:01.896 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:01Z" 2024-06-15T10:39:01.897 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:01.898 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:39:06.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:06.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:06.900 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:06.945 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:06.945 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:06.994 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:06.997+0000 7f6329635a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:06.995 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:06.997+0000 7f6329635a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:07.031 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:07.032 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:07.033 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:07.034 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:07.035 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:07Z" 2024-06-15T10:39:07.036 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:07.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:39:12.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:12.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:12.039 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:12.081 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:12.082 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:12.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:12.129+0000 7f14a3b80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:12.129 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:12.129+0000 7f14a3b80a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:12.148 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:12.149 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:12.150 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.752631_668.1", 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.752631Z", 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:12.151 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:12.152 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:12.153 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:12.153 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:12.153 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:12Z" 2024-06-15T10:39:12.153 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:12.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.752631_668.1 2024-06-15T10:39:17.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:17.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:17.155 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:17.198 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:17.199 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:17.241 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:17.241+0000 7f9eed361a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:17.242 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:17.241+0000 7f9eed361a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:17.260 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:17.260 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:17.260 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:17.260 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:17.261 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:17.262 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:17.263 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:17.264 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:17Z" 2024-06-15T10:39:17.265 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:17.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:39:22.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:22.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:22.268 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:22.313 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:22.313 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:22.365 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:22.365+0000 7f38d3ea6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:22.366 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:22.369+0000 7f38d3ea6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:22.387 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:39:22.388 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:22.389 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:22Z" 2024-06-15T10:39:22.390 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:22.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:39:27.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:27.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:27.393 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:27.438 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:27.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:27.499 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:27.501+0000 7f9dfd85ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:27.500 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:27.501+0000 7f9dfd85ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:27.523 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:27.524 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:27.525 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:27.526 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:27.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:27.528 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:27Z" 2024-06-15T10:39:27.528 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:27.529 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:39:32.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:32.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:32.530 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:32.573 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:32.574 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:32.621 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:32.621+0000 7f17b6f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:32.622 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:32.621+0000 7f17b6f07a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:32.649 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:32.649 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:32.649 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:32.649 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:32.650 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:32.651 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.266881_592.1", 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.266881Z", 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:32.652 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:32.653 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:32Z" 2024-06-15T10:39:32.654 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:32.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.266881_592.1 2024-06-15T10:39:37.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:37.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:37.656 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:37.701 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:37.701 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:37.759 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:37.761+0000 7fd9c8b00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:37.760 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:37.761+0000 7fd9c8b00a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:37.782 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:37.783 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:37.784 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:37.785 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:37.786 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:37Z" 2024-06-15T10:39:37.787 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:37.788 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:39:42.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:42.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:42.789 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:42.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:42.832 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:42.874 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:42.877+0000 7f86592dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:42.875 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:42.877+0000 7f86592dfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:42.897 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:42.898 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:42.899 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:39:42.900 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:42.901 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:42.902 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:42.903 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:42.903 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:42Z" 2024-06-15T10:39:42.903 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:42.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:39:47.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:47.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:47.905 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:47.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:47.948 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:47.990 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:47.993+0000 7fd387a6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:47.991 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:47.993+0000 7fd387a6fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:48.010 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:48.011 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:48.012 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:39:48.013 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:48.014 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:47Z" 2024-06-15T10:39:48.015 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:48.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:39:53.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:53.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:53.017 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:53.077 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:53.077 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:53.142 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:53.144+0000 7f76c86d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:53.143 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:53.144+0000 7f76c86d2a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.166 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.167 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:53.168 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:53.169 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:53.170 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:53Z" 2024-06-15T10:39:53.171 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:53.171 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:39:58.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:58.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:39:58.173 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:39:58.216 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:39:58.216 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:39:58.271 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:58.272+0000 7fda548f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:58.272 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:39:58.272+0000 7fda548f6a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:39:58.296 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:39:58.296 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:39:58.296 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:39:58.296 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:39:58.296 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:58.297 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:39:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:39:58.300 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.301 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:39:58.302 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:39:58.302 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:39:58.302 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:39:58.302 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:39:58Z" 2024-06-15T10:39:58.302 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:39:58.303 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:40:03.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:03.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:03.304 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:03.346 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:03.347 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:03.393 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:03.392+0000 7fc977d7ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:03.394 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:03.396+0000 7fc977d7ba40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:03.415 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:03.416 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.417 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:03.418 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:03.419 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:03.420 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:03.420 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:03.420 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:03Z" 2024-06-15T10:40:03.420 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:03.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:40:08.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:08.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:08.422 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:08.466 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:08.466 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:08.516 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:08.516+0000 7f1d06a5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:08.516 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:08.516+0000 7f1d06a5aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:08.535 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:08.535 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:08.535 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:08.535 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:08.536 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:08.537 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:40:08.538 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:08.539 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:08.539 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:08.539 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:40:08.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:08.542 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:08.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:08Z" 2024-06-15T10:40:08.544 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:08.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:40:13.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:13.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:13.547 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:13.590 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:13.591 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:13.634 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:13.636+0000 7f7551d9fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:13.635 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:13.636+0000 7f7551d9fa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:13.660 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:13.660 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:13.661 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:13.662 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:13.663 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:13.664 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:13.665 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:13Z" 2024-06-15T10:40:13.666 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:13.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:40:18.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:18.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:18.669 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:18.714 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:18.714 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:18.820 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:18.820+0000 7fb52752ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:18.821 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:18.820+0000 7fb52752ea40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:18.844 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:18.845 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.846 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.915314_676.1", 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.915314Z", 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:18.847 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:18.848 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:18.849 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:18.849 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:18.849 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:18Z" 2024-06-15T10:40:18.849 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:18.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.915314_676.1 2024-06-15T10:40:23.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:23.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:23.851 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:23.895 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:23.895 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:23.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:23.952+0000 7fe4e004da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:23.952 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:23.952+0000 7fe4e004da40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:23.972 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:23.973 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:23.974 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:23.975 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:23.976 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:23.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:23.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:23.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:23.980 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:23.981 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:23Z" 2024-06-15T10:40:23.982 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:23.984 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:40:28.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:28.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:28.985 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:29.028 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:29.029 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:29.072 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:29.072+0000 7f7b68fdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:29.073 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:29.072+0000 7f7b68fdda40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:29.093 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:29.094 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:29.095 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:40:29.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:29.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:29Z" 2024-06-15T10:40:29.098 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:29.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:40:34.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:34.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:34.101 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:34.144 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:34.144 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:34.198 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:34.196+0000 7f3e2d3f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:34.199 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:34.200+0000 7f3e2d3f9a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:34.217 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:34.217 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:34.218 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:34.219 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:34.220 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:34.221 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:34Z" 2024-06-15T10:40:34.222 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:34.223 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:40:39.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:39.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:39.224 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:39.269 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:39.270 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:39.321 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:39.320+0000 7f695f620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:39.322 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:39.324+0000 7f695f620a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:39.345 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:39.346 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:39.347 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.348 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:39.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:39.350 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:39Z" 2024-06-15T10:40:39.351 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:39.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:40:44.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:44.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:44.355 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:44.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:44.399 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:44.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:44.439+0000 7f9a3f99aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:44.442 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:44.443+0000 7f9a3f99aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:44.461 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:44.462 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:44.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:44.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:44.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:44.466 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:44Z" 2024-06-15T10:40:44.466 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:44.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:40:49.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:49.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:49.468 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:49.511 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:49.511 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:49.554 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:49.551+0000 7fbd1f169a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:49.555 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:49.555+0000 7fbd1f169a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:49.573 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:49.574 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:49.575 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:49.576 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:49.577 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:49Z" 2024-06-15T10:40:49.578 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:49.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:40:54.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:54.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:54.580 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:54.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:54.625 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:54.680 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:54.679+0000 7f1884760a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:54.681 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:54.679+0000 7f1884760a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:54.701 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:54.702 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:54.703 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:40:54.704 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:54.705 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:54.706 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:54Z" 2024-06-15T10:40:54.707 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:54.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:40:59.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:59.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:40:59.709 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:40:59.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:40:59.747 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:40:59.800 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:59.799+0000 7ff82ada3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:59.801 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:40:59.799+0000 7ff82ada3a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:40:59.821 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:40:59.822 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:40:59.823 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:59.824 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:40:59.825 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:40:59Z" 2024-06-15T10:40:59.826 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:40:59.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:41:04.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:04.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:04.830 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:04.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:04.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:04.918 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:04.915+0000 7f77492bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:04.918 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:04.919+0000 7f77492bfa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:04.937 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:04.938 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:04.939 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.799732_672.1", 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.799732Z", 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.940 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:04.941 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:04.942 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:04.942 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:04.942 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:04Z" 2024-06-15T10:41:04.942 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:04.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.799732_672.1 2024-06-15T10:41:09.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:09.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:09.945 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:09.987 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:09.988 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:10.043 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:10.043+0000 7f55502e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:10.044 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:10.043+0000 7f55502e7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:10.065 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:10.066 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:10.067 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.068 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.069 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:10.070 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:10.070 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:10.070 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:10.070 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:10Z" 2024-06-15T10:41:10.070 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:10.071 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:41:15.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:15.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:15.073 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:15.115 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:15.116 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:15.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:15.159+0000 7fa65ff84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:15.161 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:15.159+0000 7fa65ff84a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:15.181 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:15.181 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:15.181 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:15.181 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:15.182 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.183 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:15.184 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:41:15.185 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:15.186 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:15.187 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:15.188 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:15.188 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:15.188 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:15Z" 2024-06-15T10:41:15.188 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:15.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:41:20.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:20.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:20.190 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:20.233 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:20.233 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:20.295 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:20.295+0000 7f43abc66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:20.296 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:20.295+0000 7f43abc66a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:20.321 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:20.322 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:20.323 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.152646_587.1", 2024-06-15T10:41:20.324 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.152646Z", 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:20.325 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:20Z" 2024-06-15T10:41:20.326 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:20.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.152646_587.1 2024-06-15T10:41:25.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:25.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:25.329 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:25.373 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:25.373 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:25.416 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:25.415+0000 7f85e8851a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:25.417 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:25.415+0000 7f85e8851a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:25.437 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:25.438 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:25.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.438 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:25.438 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:25.438 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:25.439 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:25.440 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:25.441 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:25.441 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:25.441 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:25.441 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:25.441 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.442 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:25.443 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:25.443 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:25.443 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:25.443 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:41:25.443 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:25.444 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:25.444 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:25.444 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:41:25.444 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:25.444 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:25.445 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:25.446 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:25.447 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:25.448 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:25.448 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:25Z" 2024-06-15T10:41:25.448 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:25.449 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:41:30.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:30.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:30.451 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:30.495 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:30.495 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:30.537 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:30.535+0000 7fdcdd906a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:30.538 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:30.539+0000 7fdcdd906a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:30.556 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:30.558 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:30.559 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:30.560 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:30.561 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:30Z" 2024-06-15T10:41:30.562 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:30.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:41:35.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:35.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:35.564 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:35.607 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:35.607 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:35.665 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:35.662+0000 7faf0e658a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:35.666 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:35.662+0000 7faf0e658a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:35.692 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:35.693 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:35.694 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.695 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:35.695 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438845.760139_669.1", 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:07:25.760139Z", 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:35.696 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:35.697 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:35.698 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:35.698 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:35Z" 2024-06-15T10:41:35.698 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:35.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438845.760139_669.1 2024-06-15T10:41:40.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:40.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:40.700 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:40.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:40.743 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:40.803 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:40.802+0000 7f35175f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:40.805 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:40.802+0000 7f35175f7a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:40.823 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:40.824 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:40.825 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:40.826 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:40.827 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:40.828 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:40.829 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:40Z" 2024-06-15T10:41:40.829 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:40.830 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:41:45.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:45.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:45.831 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:45.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:45.874 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:45.937 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:45.934+0000 7feadd6afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:45.938 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:45.938+0000 7feadd6afa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:45.963 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:45.964 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:45.965 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:45.966 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:45Z" 2024-06-15T10:41:45.967 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:45.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:41:50.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:50.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:50.970 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:51.013 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:51.013 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:51.065 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:51.062+0000 7fc1b6b3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:51.066 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:51.062+0000 7fc1b6b3ca40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:51.094 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:51.095 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:51.096 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:51.097 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.098 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:51.099 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:51Z" 2024-06-15T10:41:51.100 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:51.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:41:56.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:56.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:41:56.102 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:41:56.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:41:56.146 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:41:56.192 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:56.190+0000 7f6deb144a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:56.193 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:41:56.190+0000 7f6deb144a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:41:56.213 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:41:56.213 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:41:56.213 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:41:56.213 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:56.214 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:56.215 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.268091_593.1", 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.268091Z", 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.216 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.217 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:41:56.218 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:41:56.218 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:41:56.218 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:41:56.218 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:41:56Z" 2024-06-15T10:41:56.218 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:41:56.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.268091_593.1 2024-06-15T10:42:01.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:01.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:01.220 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:01.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:01.264 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:01.317 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:01.314+0000 7f67d3257a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:01.318 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:01.318+0000 7f67d3257a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:01.337 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:01.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:01.337 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:01.338 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:01.339 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:01.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:01.341 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:01.342 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:01.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:01.342 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:01Z" 2024-06-15T10:42:01.342 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:01.343 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:42:06.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:06.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:06.344 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:06.388 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:06.388 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:06.440 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:06.438+0000 7fc32116aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:06.441 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:06.438+0000 7fc32116aa40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.462 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:06.463 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:06.464 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.163871_589.1", 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:06.465 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.163871Z", 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:06.466 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:06Z" 2024-06-15T10:42:06.467 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:06.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.163871_589.1 2024-06-15T10:42:11.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:11.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:11.470 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:11.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:11.514 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:11.578 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:11.574+0000 7f4ddfc38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:11.579 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:11.578+0000 7f4ddfc38a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:11.601 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.602 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:11.603 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.179989_590.1", 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.179989Z", 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:11.604 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:11Z" 2024-06-15T10:42:11.605 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:11.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.179989_590.1 2024-06-15T10:42:16.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:16.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:16.607 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:16.650 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:16.650 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:16.698 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:16.694+0000 7fd4f8625a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:16.699 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:16.698+0000 7fd4f8625a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:16.719 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:16.720 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:42:16.721 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:16Z" 2024-06-15T10:42:16.723 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:16.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:42:21.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:21.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:21.725 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:21.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:21.770 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:21.821 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:21.822+0000 7fb883d54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:21.822 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:21.826+0000 7fb883d54a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:21.861 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:21.861 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:21.861 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:21.861 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:21.862 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:21.863 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.864 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.865 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:21.866 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:21.866 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:21.866 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:21.866 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:21Z" 2024-06-15T10:42:21.866 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:21.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:42:26.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:26.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-06-15T10:42:26.868 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-06-15T10:42:26.912 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2024-06-15T10:42:26.912 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2024-06-15T10:42:26.955 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:26.957+0000 7fa55f766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:26.956 INFO:teuthology.orchestra.run.smithi131.stderr:2024-06-15T10:42:26.957+0000 7fa55f766a40 20 RGW-SYNC:meta: read sync status 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "6634f146-140c-4941-81ef-ff44e7855a30", 2024-06-15T10:42:26.976 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718437090.276409_28.1", 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T07:38:10.276409Z", 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.977 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.930258_623.1", 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.930258Z", 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2024-06-15T10:42:26.978 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718438539.261071_591.1", 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T08:02:19.261071Z", 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1718443838.978738_1039.1", 2024-06-15T10:42:26.979 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2024-06-15T09:30:38.978738Z", 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2024-06-15T10:42:26Z" 2024-06-15T10:42:26.980 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-06-15T10:42:26.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1718445381.083679_1521.1 target=1_1718438539.261071_591.1 2024-06-15T10:42:31.983 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... FAIL 2024-06-15T10:42:31.983 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/rgw_multisite.py", line 197, in admin 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:AssertionError 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.984 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.985 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_period_incremental_sync 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 933, in test_multi_period_incremental_sync 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.986 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.987 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1061, in test_multi_zone_redirect 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.988 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1090, in test_zonegroup_remove 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.989 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1173, in test_bucket_sync_disable 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.990 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1197, in test_bucket_sync_enable_right_after_disable 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.991 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1232, in test_bucket_sync_disable_enable 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.992 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1276, in test_multipart_object_sync 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.993 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_encrypted_object_sync 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1313, in test_encrypted_object_sync 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.994 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1343, in test_bucket_index_log_trim 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-06-15T10:42:31.995 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1338, in make_test_bucket 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.996 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1397, in test_bucket_reshard_index_log_trim 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1392, in make_test_bucket 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.997 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1474, in test_bucket_reshard_incremental 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.998 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1504, in test_bucket_reshard_full 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:31.999 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1537, in test_bucket_creation_time 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.000 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1622, in test_bucket_sync_run_basic_incremental 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1685, in test_zap_init_bucket_sync_run 2024-06-15T10:42:32.001 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_role_sync 2024-06-15T10:42:32.002 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1732, in test_role_sync 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.003 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1750, in test_bucket_full_sync_after_data_sync_init 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1782, in test_resharded_bucket_full_sync_after_data_sync_init 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1821, in test_bucket_incremental_sync_after_data_sync_init 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1859, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-06-15T10:42:32.006 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 1907, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_policy_config_zonegroup: 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.008 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2097, in test_sync_policy_config_zonegroup 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.009 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2169, in test_sync_flow_symmetrical_zonegroup_all 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.011 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_select: 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2232, in test_sync_flow_symmetrical_zonegroup_select 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.012 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_directional_zonegroup_select: 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2302, in test_sync_flow_directional_zonegroup_select 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.013 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2415, in test_sync_single_bucket 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.014 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_different_buckets: 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2532, in test_sync_different_buckets 2024-06-15T10:42:32.015 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.016 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2678, in test_sync_multiple_buckets_to_single 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket_to_multiple: 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-06-15T10:42:32.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 2799, in test_sync_single_bucket_to_multiple 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 10943.679s 2024-06-15T10:42:32.018 INFO:tasks.rgw_multisite_tests: 2024-06-15T10:42:32.019 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=1, failures=32) 2024-06-15T10:42:32.019 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-06-15T10:42:32.272 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=999d438a3ab1466d9694c71cfdab404d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-06-15T10:42:32.273 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-06-15T10:42:32.287 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-06-15T10:42:32.330 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-06-15T10:42:32.363 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-06-15T10:42:32.363 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:42:32.375 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T10:42:32.363+0000 7fe2ec09f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 29216) UID: 0 2024-06-15T10:42:32.375 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2024-06-15T10:42:32.363+0000 7fe2ef8e6a40 -1 shutting down 2024-06-15T10:42:33.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-06-15T10:42:38.464 INFO:tasks.rgw.c1.client.0:Stopped 2024-06-15T10:42:38.465 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-06-15T10:42:38.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/vault-root-token 2024-06-15T10:42:38.649 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-06-15T10:42:38.649 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:42:38.650 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T10:42:38.647+0000 7fcb849d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 30417) UID: 0 2024-06-15T10:42:38.650 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2024-06-15T10:42:38.647+0000 7fcb8821da40 -1 shutting down 2024-06-15T10:42:38.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-06-15T10:42:44.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-06-15T10:42:44.751 INFO:tasks.rgw.c1.client.1:Stopped 2024-06-15T10:42:44.751 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-06-15T10:42:44.758 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/vault-root-token 2024-06-15T10:42:44.813 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-06-15T10:42:44.813 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:42:44.898 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T10:42:44.817+0000 7fa2f5c5b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 29018) UID: 0 2024-06-15T10:42:44.902 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2024-06-15T10:42:44.817+0000 7fa2f94a2a40 -1 shutting down 2024-06-15T10:42:49.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-06-15T10:42:50.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-06-15T10:42:50.915 INFO:tasks.rgw.c2.client.0:Stopped 2024-06-15T10:42:50.915 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-06-15T10:42:50.932 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/vault-root-token 2024-06-15T10:42:51.089 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-06-15T10:42:51.089 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:42:51.090 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2024-06-15T10:42:51.093+0000 7fe53680c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 27405) UID: 0 2024-06-15T10:42:51.090 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2024-06-15T10:42:51.093+0000 7fe53a053a40 -1 shutting down 2024-06-15T10:42:55.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-06-15T10:42:55.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-06-15T10:42:57.191 INFO:tasks.rgw.c2.client.1:Stopped 2024-06-15T10:42:57.192 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-06-15T10:42:57.198 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/vault-root-token 2024-06-15T10:42:57.255 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-15T10:42:57.265 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-06-15T10:42:57.878 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-15T10:42:57.878 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-15T10:42:57.878 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-06-15T10:42:57.879 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:01.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~0s 2024-06-15T10:43:01.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-06-15T10:43:03.981 INFO:tasks.ceph.c2.osd.0:Stopped 2024-06-15T10:43:03.981 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-06-15T10:43:03.982 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:06.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2024-06-15T10:43:06.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 has been restored 2024-06-15T10:43:10.083 INFO:tasks.ceph.c2.osd.1:Stopped 2024-06-15T10:43:10.084 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-06-15T10:43:10.084 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:11.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~0s 2024-06-15T10:43:11.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 has been restored 2024-06-15T10:43:16.187 INFO:tasks.ceph.c2.osd.2:Stopped 2024-06-15T10:43:16.187 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-15T10:43:16.187 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-06-15T10:43:16.188 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:16.219 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-06-15T10:43:16.219 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-15T10:43:16.220 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-06-15T10:43:16.220 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:16.290 INFO:tasks.ceph.c2.mon.a:Stopped 2024-06-15T10:43:16.291 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-15T10:43:16.291 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-06-15T10:43:16.407 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T10:43:16.407 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-06-15T10:43:16.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T10:43:16.698 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-06-15T10:43:16.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 has been restored 2024-06-15T10:43:16.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T10:43:16.919 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-06-15T10:43:17.130 INFO:tasks.ceph:Archiving mon data... 2024-06-15T10:43:17.131 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/data/c2.mon.a.tgz 2024-06-15T10:43:17.131 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-06-15T10:43:17.142 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.mR5RWE3Kvm 2024-06-15T10:43:17.143 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.mR5RWE3Kvm 2024-06-15T10:43:17.323 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.mR5RWE3Kvm 2024-06-15T10:43:17.419 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.mR5RWE3Kvm is 451KB 2024-06-15T10:43:17.477 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.mR5RWE3Kvm 2024-06-15T10:43:17.483 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-15T10:43:17.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-06-15T10:43:17.486 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-06-15T10:43:17.541 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-15T10:43:17.553 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-06-15T10:43:18.065 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-15T10:43:18.065 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-15T10:43:18.066 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-06-15T10:43:18.066 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:18.160 INFO:tasks.ceph.c1.osd.0:Stopped 2024-06-15T10:43:18.161 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-06-15T10:43:18.161 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:22.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2024-06-15T10:43:24.262 INFO:tasks.ceph.c1.osd.1:Stopped 2024-06-15T10:43:24.263 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-06-15T10:43:24.263 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:27.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2024-06-15T10:43:27.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 has been restored 2024-06-15T10:43:30.365 INFO:tasks.ceph.c1.osd.2:Stopped 2024-06-15T10:43:30.365 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-15T10:43:30.366 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-06-15T10:43:30.366 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:30.398 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-06-15T10:43:30.398 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-15T10:43:30.398 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-06-15T10:43:30.399 INFO:teuthology.orchestra.run:waiting for 300 2024-06-15T10:43:32.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-06-15T10:43:32.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 has been restored 2024-06-15T10:43:36.501 INFO:tasks.ceph.c1.mon.a:Stopped 2024-06-15T10:43:36.501 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-15T10:43:36.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-06-15T10:43:36.597 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T10:43:36.597 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-06-15T10:43:36.886 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T10:43:36.887 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-06-15T10:43:37.150 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T10:43:37.151 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-06-15T10:43:37.361 INFO:tasks.ceph:Archiving mon data... 2024-06-15T10:43:37.362 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/data/c1.mon.a.tgz 2024-06-15T10:43:37.365 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-06-15T10:43:37.379 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.B5Cadp42a1 2024-06-15T10:43:37.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.B5Cadp42a1 2024-06-15T10:43:37.609 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.B5Cadp42a1 2024-06-15T10:43:37.700 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.B5Cadp42a1 is 520KB 2024-06-15T10:43:37.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a has been restored 2024-06-15T10:43:37.761 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.B5Cadp42a1 2024-06-15T10:43:37.767 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-15T10:43:37.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-06-15T10:43:37.813 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-06-15T10:43:37.827 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-06-15T10:43:37.827 DEBUG:teuthology.orchestra.run.smithi012:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-06-15T10:43:37.868 DEBUG:teuthology.orchestra.run.smithi131:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-06-15T10:43:37.897 INFO:teuthology.orchestra.run.smithi012.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-06-15T10:43:37.990 INFO:teuthology.orchestra.run.smithi131.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-06-15T10:43:37.994 INFO:tasks.ceph:Archiving crash dumps... 2024-06-15T10:43:37.996 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi012/crash 2024-06-15T10:43:37.997 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-15T10:43:38.012 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi131/crash 2024-06-15T10:43:38.013 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-15T10:43:38.025 INFO:tasks.ceph:Compressing logs... 2024-06-15T10:43:38.026 DEBUG:teuthology.orchestra.run.smithi012:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T10:43:38.061 DEBUG:teuthology.orchestra.run.smithi131:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T11:10:09.878 INFO:tasks.ceph:Archiving logs... 2024-06-15T11:10:09.900 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi012/log 2024-06-15T11:10:09.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T11:12:27.289 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi131/log 2024-06-15T11:12:27.290 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T11:13:05.849 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-15T11:13:05.860 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-15T11:13:05.861 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T11:13:05.864 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T11:13:05.959 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-15T11:13:05.969 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-15T11:13:05.969 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T11:13:05.972 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T11:13:06.016 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T11:13:06.016 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2024-06-15T11:13:06.017 INFO:teuthology.orchestra.run.smithi012.stdout:+hv01.front.sepi 67.205.162.81 3 u 30 512 377 0.234 -1.549 0.287 2024-06-15T11:13:06.017 INFO:teuthology.orchestra.run.smithi012.stdout:*hv02.front.sepi 63.231.80.2 3 u 8 512 377 0.184 +2.330 0.362 2024-06-15T11:13:06.017 INFO:teuthology.orchestra.run.smithi012.stdout:+hv03.front.sepi 74.6.168.72 3 u 476 512 377 0.153 +0.900 0.226 2024-06-15T11:13:06.017 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================== 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout:*hv01.front.sepi 67.205.162.81 3 u 10 512 377 0.176 -2.938 0.159 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout:+hv02.front.sepi 63.231.80.2 3 u 413 512 377 0.141 +1.247 0.340 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout:+hv03.front.sepi 74.6.168.72 3 u 415 512 377 0.179 -0.148 0.192 2024-06-15T11:13:06.042 INFO:teuthology.orchestra.run.smithi131.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-15T11:13:06.043 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-15T11:13:06.053 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-15T11:13:06.054 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-15T11:13:06.076 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-15T11:13:06.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-15T11:13:06.141 INFO:teuthology.task.internal:Duration was 13571.761447 seconds 2024-06-15T11:13:06.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-15T11:13:06.175 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-15T11:13:06.175 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T11:13:06.180 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T11:13:06.462 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-15T11:13:06.463 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-06-15T11:13:06.463 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T11:13:06.506 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-06-15T11:13:06.506 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T11:13:06.518 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-15T11:13:06.518 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T11:13:06.549 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T11:13:06.635 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-15T11:13:06.636 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T11:13:06.638 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T11:13:06.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-15T11:13:06.904 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-15T11:13:06.905 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T11:13:06.942 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T11:13:06.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-15T11:13:06.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:13:06.998 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:13:07.038 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-06-15T11:13:07.050 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-06-15T11:13:07.087 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:13:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T11:13:07.112 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:13:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T11:13:07.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-15T11:13:07.145 INFO:teuthology.task.internal:Transferring archived files... 2024-06-15T11:13:07.146 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi012 2024-06-15T11:13:07.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T11:13:07.177 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:51:11-rgw-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756785/remote/smithi131 2024-06-15T11:13:07.177 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T11:13:07.200 INFO:teuthology.task.internal:Removing archive directory... 2024-06-15T11:13:07.200 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T11:13:07.217 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T11:13:07.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-15T11:13:07.258 INFO:teuthology.task.internal:Not uploading archives. 2024-06-15T11:13:07.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-15T11:13:07.285 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-15T11:13:07.285 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T11:13:07.287 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T11:13:07.292 INFO:teuthology.orchestra.run.smithi012.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 11:13 /home/ubuntu/cephtest 2024-06-15T11:13:07.298 INFO:teuthology.orchestra.run.smithi131.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 11:13 /home/ubuntu/cephtest 2024-06-15T11:13:07.298 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-15T11:13:07.318 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-15T11:13:07.387 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} duration: 13571.761447191238 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=999d438a3ab1466d9694c71cfdab404d status: fail success: false 2024-06-15T11:13:07.387 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T11:13:07.463 INFO:teuthology.run:FAIL